From: Nicholas Bellinger <nab@xxxxxxxxxxxxxxx> Hi folks, This series is the second RFC for iSCSI Extensions for RDMA (ISER) target support with existing iscsi-target TCP based socket code, planned for a future v3.10 merge. This includes a basic iscsit_transport API that allows different transports to reside under a single iscsi-target configfs control plane, using an pre-defined network portal attribute to enable a rdma_cm listener on top of existing ipoib portals. This code is available in git against v3.9-rc3 here: git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending.git iser_target-rfc-v2 Thanks to Or Gerlitz, Roland Dreier, Asias He and Andy Grover for their -v1 review comments. The full changelog since RFC-v1 have been included into the individual patches. The highlights for drivers/target/iscsi/ code include: - Add ->iscsit_queue_data_in() to remove extra context switch on RDMA_WRITE - Add ->iscsit_queue_status() to remove extra context switch on IB_SEND status - Add ->iscsit_get_dataout() to remove extra context switch on RDMA_READ - Drop ->iscsit_free_cmd() - Drop ->iscsit_unmap_cmd() - Disable iscsit_ack_from_expstatsn() usage for RDMAExtentions=Yes - Disable iscsit_allocate_datain_req() usage for RDMAExtentions=Yes - Add target_get_sess_cmd() reference counting to iscsit_setup_scsi_cmd() - Add TFO->lio_check_stop_free() fabric API caller - Convert existing usage of iscsit_build_r2ts_for_cmd() to ->iscsit_get_dataout() - Drop RDMAExtentions=Yes specific check in iscsit_build_r2ts_for_cmd() - Add iscsit_queue_rsp() for iscsit_transport->iscsit_queue_data_in() and iscsit_transport->iscsit_queue_status() - Update lio_queue_data_in() to use ->iscsit_queue_data_in() - Update lio_queue_status() to use ->iscsit_queue_status() and for drivers/infiniband/ulp/isert/ code: - Drop isert_cmd->cmd_kref in favor of se_cmd->cmd_kref usage - Add struct isert_device in order to support multiple EQs + CQ pooling - Drop tasklets and cqs from isert_conn - Bump ISERT_MAX_CQ to 64 - Update isert_conn_setup_qp() to assign cq based upon least used - Add isert_create_device_ib_res() to setup PD, CQs and MRs for each underlying struct ib_device, instead of using per isert_conn resources. - Add isert_free_device_ib_res() to release PD, CQs and MRs for each underlying struct ib_device. - Change isert_connect_request() to drop PD, CQs and MRs allocation, and use isert_device_find_by_ib_dev() instead. - Change isert_connect_release() to decrement cq_active_qps, and drop PD, CQs and MRs resource release. - Make isert_create_device_ib_res() determine device->cqs_used based upon num_online_cpus() - Change isert_put_cmd() to perform per iscsi_opcode specific release logic - Add isert_unmap_cmd() call for ISCSI_OP_SCSI_CMD from isert_put_cmd() - Drop ISTATE_REMOVE processing from isert_immediate_queue() - Drop ISTATE_SEND_DATAIN processing from isert_response_queue() - Drop ISTATE_SEND_STATUS processing from isert_response_queue() - Drop iscsit_transport->iscsit_unmap_cmd() and ->iscsit_free_cmd() - Drop ISTATE_SEND_R2T processing from isert_immediate_queue() As before, review patches are broken down into: Patch #1 adds the export of target_get_sess_cmd to be used by iscsi-target Patch #2 -> #4 include iscsi-target API template, conversion of iscsi/tcp login path to use API template, plus add iser RFC parameter keys. Patch #5 -> #6 allow external iscsi_cmd descriptor allocation / free, and refactoring of RX side PDU request handling to allow incoming PDU logic to be called by external ib_isert workqueue process context. Patch #7 allows iscsi-target to use per transport API template immediate / response callbacks in the per-connection TX thread completion path, and refactoring of response PDU creation for export to external ib_isert code. Patch #8 adds the pre-defined iser network portal attribute under the existing iscsi-target configfs tree. Patch #9 -> #12 is the external ib_isert.ko module code seperated into individual commits for review. So at this point this code is functional and pushing sustained RDMA_WRITE + RDMA_READ traffic using open-iscsi on top of multiple iser network portals + multiple IB HCA ports + multiple LUNs. Thus far we're using Mellanox IB HCAs for initial development, and will be verfiying using RCoE capable NICs as well in the near future. Many thanks again to Or Gerlitz and Yaron Haviv from Mellanox for their support. --nab Nicholas Bellinger (12): target: Add export of target_get_sess_cmd symbol iscsi-target: Add iscsit_transport API template iscsi-target: Initial traditional TCP conversion to iscsit_transport iscsi-target: Add iser-target parameter keys + setup during login iscsi-target: Add per transport iscsi_cmd alloc/free iscsi-target: Refactor RX PDU logic + export request PDU handling iscsi-target: Refactor TX queue logic + export response PDU creation iscsi-target: Add iser network portal attribute iser-target: Add base + proto includes iser-target: Add logic for verbs iser-target: Add logic for core iser-target: Add Makefile + Kconfig drivers/infiniband/Kconfig | 1 + drivers/infiniband/Makefile | 1 + drivers/infiniband/ulp/isert/Kconfig | 6 + drivers/infiniband/ulp/isert/Makefile | 5 + drivers/infiniband/ulp/isert/isert_base.h | 142 ++ drivers/infiniband/ulp/isert/isert_core.c | 1730 ++++++++++++++++++++++++ drivers/infiniband/ulp/isert/isert_core.h | 11 + drivers/infiniband/ulp/isert/isert_proto.h | 47 + drivers/infiniband/ulp/isert/isert_verbs.c | 594 ++++++++ drivers/infiniband/ulp/isert/isert_verbs.h | 5 + drivers/target/iscsi/Makefile | 3 +- drivers/target/iscsi/iscsi_target.c | 1132 +++++++++------- drivers/target/iscsi/iscsi_target.h | 3 +- drivers/target/iscsi/iscsi_target_configfs.c | 94 ++- drivers/target/iscsi/iscsi_target_core.h | 26 +- drivers/target/iscsi/iscsi_target_device.c | 7 +- drivers/target/iscsi/iscsi_target_erl1.c | 13 +- drivers/target/iscsi/iscsi_target_login.c | 468 +++++-- drivers/target/iscsi/iscsi_target_login.h | 6 + drivers/target/iscsi/iscsi_target_nego.c | 182 +--- drivers/target/iscsi/iscsi_target_nego.h | 11 +- drivers/target/iscsi/iscsi_target_parameters.c | 87 ++- drivers/target/iscsi/iscsi_target_parameters.h | 16 +- drivers/target/iscsi/iscsi_target_tmr.c | 4 +- drivers/target/iscsi/iscsi_target_tpg.c | 6 +- drivers/target/iscsi/iscsi_target_transport.c | 55 + drivers/target/iscsi/iscsi_target_util.c | 53 +- drivers/target/iscsi/iscsi_target_util.h | 1 + drivers/target/target_core_transport.c | 4 +- include/target/iscsi/iscsi_transport.h | 81 ++ include/target/target_core_fabric.h | 2 +- 31 files changed, 3981 insertions(+), 815 deletions(-) create mode 100644 drivers/infiniband/ulp/isert/Kconfig create mode 100644 drivers/infiniband/ulp/isert/Makefile create mode 100644 drivers/infiniband/ulp/isert/isert_base.h create mode 100644 drivers/infiniband/ulp/isert/isert_core.c create mode 100644 drivers/infiniband/ulp/isert/isert_core.h create mode 100644 drivers/infiniband/ulp/isert/isert_proto.h create mode 100644 drivers/infiniband/ulp/isert/isert_verbs.c create mode 100644 drivers/infiniband/ulp/isert/isert_verbs.h create mode 100644 drivers/target/iscsi/iscsi_target_transport.c create mode 100644 include/target/iscsi/iscsi_transport.h -- 1.7.2.5 -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
