On Sun, 2013-06-23 at 17:47 +0300, Or Gerlitz wrote: > On 23/06/2013 17:40, Or Gerlitz wrote: > > > > > > there you go, here's the output > > > > isert_cma_handler: event 4 status 0 conn ffff88011a55d600 id > > ffff8801085c5400 > > RDMA_CM_EVENT_CONNECT_REQUEST: >>>>>>>>>>>>>>> > > Entering isert_connect_request cma_id: ffff8801085c5400, context: > > ffff88011a55d600 > > Using responder_resources: 1 initiator_depth: 4 > > Set login_buf: ffff880116a38000 login_req_buf: ffff880116a38000 > > login_rsp_buf: ffff880116a3a000 > > Using 3 CQs, device mlx4_0 supports 3 vectors > [...] > > and here's the initiator output (if you set debug_level=3 for ib_iser) > > > iser: iser_rcv_completion:op 0x3f itt 0xffffffff dlen 0 > > iser: iscsi_iser_recv:wrong datalen 48 (hdr), 0 (IB) > > > > This means that LIO sent ISCSI_OP_REJECT (0x3f) and that there's a bug > for reject since the iSCSI header > says there are 48 bytes beyond the BHS but on the wire [1] there where none > Hi Or & Co, Ok, sendtargets discovery is now functioning over iser. The updated target patches + v2 changelogs are in for-next commits here: http://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/commit/?h=for-next&id=889c8a68b8483a8b3482ac440af3ad7368c58555 http://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/commit/?h=for-next&id=4ebc2e95f49aca3114acb13b15c3e6f769ee6300 http://git.kernel.org/cgit/linux/kernel/git/nab/target-pending.git/commit/?h=for-next&id=58203ef7ae06f17f7ee238491cd7301abe3dfc19 The running initiator + target discovery output is below. Please go ahead and give it a shot on your setup. One thing to note, this currently *only* works with open-iscsi when the TEXT_RSP payload posted by the target is <= 128 bytes (eg: equal to or less than the initiator posted ISER_RECV_DATA_SEG_LEN), otherwise a IB_WC_LOC_LEN_ERR error will be generated by the initiator.. So that said, I think we need to go ahead and bump the open-iscsi side ISER_RECV_DATA_SEG_LEN constant to 8192 in order to match what is actually advertised by the initiator's MaxRecvDataSegmentLength during discovery negotiation. Or and Mike, here is a quick patch on the initiator side.. Ideally this would follow the negotiated MRDSL, but it's certainly better than hitting IB_WC_LOC_LEN_ERR errors during (most) typical sendtargets cases. WDYT..? diff --git a/drivers/infiniband/ulp/iser/iscsi_iser.h b/drivers/infiniband/ulp/iser/iscsi_iser.h index 4f069c0..2fab32c 100644 --- a/drivers/infiniband/ulp/iser/iscsi_iser.h +++ b/drivers/infiniband/ulp/iser/iscsi_iser.h @@ -153,7 +153,7 @@ struct iser_cm_hdr { /* Constant PDU lengths calculations */ #define ISER_HEADERS_LEN (sizeof(struct iser_hdr) + sizeof(struct iscsi_hdr)) -#define ISER_RECV_DATA_SEG_LEN 128 +#define ISER_RECV_DATA_SEG_LEN 8192 #define ISER_RX_PAYLOAD_SIZE (ISER_HEADERS_LEN + ISER_RECV_DATA_SEG_LEN) #define ISER_RX_LOGIN_SIZE (ISER_HEADERS_LEN + ISCSI_DEF_MAX_RECV_SEG_LEN) Also, as you reported above there is still a problem somewhere with the posting/handling of REJECTs PDUs. I'm currently tracking this down, and is a bug separate from the v3.11 changes. More on that soon.. --nab Initiator side: [ 251.926481] iscsi: registered transport (iser) [ 251.926705] caps for iscsi transport: 0x00000089 [ 296.946755] iser: iser_connect:connecting to: 10.100.0.1, port 0xbd0c [ 296.951793] iser: iscsi_iser_ep_poll:ib conn ffff8802354699b8 rc = 0 [ 296.952319] iser: iser_cma_handler:event 0 status 0 conn ffff8802354699b8 id ffff880234325c00 [ 296.952737] iser: iser_create_device_ib_res:using 4 CQs, device mlx4_0 supports 19 vectors [ 296.968061] iser: iser_cma_handler:event 2 status 0 conn ffff8802354699b8 id ffff880234325c00 [ 296.978296] iser: iser_create_ib_conn_res:cq index 0 used for ib_conn ffff8802354699b8 [ 296.979763] iser: iser_create_ib_conn_res:setting conn ffff8802354699b8 cma_id ffff880234325c00: fmr_pool ffff880236a23640 qp ffff8802346c6e00 [ 297.011751] iser: iser_cma_handler:event 9 status 0 conn ffff8802354699b8 id ffff880234325c00 [ 297.954147] iser: iscsi_iser_ep_poll:ib conn ffff8802354699b8 rc = 1 [ 297.954614] scsi6 : iSCSI Initiator over iSER, v.0.1 [ 297.955350] iser: iscsi_iser_conn_bind:binding iscsi/iser conn ffff88022b7fadf0 ffff88022b7fafd0 to ib_conn ffff8802354699b8 [ 297.955807] iser: iscsi_iser_mtask_xmit:mtask xmit [cid 0 itt 0x0] [ 297.955810] iser: iser_post_rx_bufs:req op 43 flags 87 [ 297.955811] iser: iser_post_rx_bufs:Initially post: 32 [ 297.966140] iser: iser_rcv_completion:op 0x23 itt 0x0 dlen 176 [ 297.966143] iser: iscsi_iser_recv:aligned datalen (173) hdr, 176 (IB) [ 297.966152] iser: iser_cq_tasklet_fn:got 1 rx 1 tx completions [ 297.966576] iser: iscsi_iser_mtask_xmit:mtask xmit [cid 0 itt 0x0] [ 297.966581] iser: iser_post_rx_bufs:req op 4 flags 80 [ 297.982123] iser: iser_rcv_completion:op 0x24 itt 0x0 dlen 84 [ 297.982126] iser: iscsi_iser_recv:aligned datalen (83) hdr, 84 (IB) [ 297.982135] iser: iser_cq_tasklet_fn:got 1 rx 1 tx completions [ 297.983249] iser: iscsi_iser_ep_disconnect:ib conn ffff8802354699b8 state 2 [ 297.983741] iser: iser_cq_tasklet_fn:got 64 rx 0 tx completions and Target side: [ 265.227096] devattr->max_sge_rd: 0 [ 265.227102] isert_conn_setup_qp: Using min_index: 0 [ 265.227103] isert_conn_setup_qp cma_id->device: ffff88045cd90000 [ 265.227105] isert_conn_setup_qp conn_pd->device: ffff88045cd90000 [ 265.227931] rdma_create_qp() returned success >>>>>>>>>>>>>>>>>>>>>>>>>. [ 265.227934] isert_connect_request() waking up np_accept_wq: ffff880455aa9ff0 [ 265.227944] Setup sge: addr: 45ce30000 length: 8268 0x00002d00 [ 265.227948] ib_post_recv(): returned success >>>>>>>>>>>>>>>>>>>>>>>> [ 265.227951] Before rdma_accept >>>>>>>>>>>>>>>>>>>>. [ 265.228123] After rdma_accept >>>>>>>>>>>>>>>>>>>>>. [ 265.228127] Processing isert_accept_np: isert_conn: ffff88045cf8f3f0 [ 265.228156] Added timeout timer to iSCSI login request for 15 seconds. [ 265.228158] Moving to TARG_CONN_STATE_XPT_UP. [ 265.228160] isert_get_login_rx before conn_login_comp conn: ffff88045d1e48c8 [ 265.229301] isert_cma_handler: event 9 status 0 conn ffff88045cf8f3f0 id ffff88045d1e4480 [ 265.229303] RDMA_CM_EVENT_ESTABLISHED >>>>>>>>>>>>>> [ 266.175652] ISER login_buf: Using rx_dma: 0x45ce30000, rx_buflen: 8268 [ 266.175656] iSCSI opcode: 0x43, ITT: 0x00000000, flags: 0x87 dlen: 244 [ 266.175660] Using login payload size: 244, rx_buflen: 244 MAX_KEY_VALUE_PAIRS: 8192 [ 266.175664] iSERT: Decremented post_recv_buf_count: 0 [ 266.175668] isert_get_login_rx processing login->req: ffff8808573a16d4 [ 266.175672] Received iSCSI login request from 10.100.0.5 on IB/iSER Network Portal 10.100.0.1:3261 [ 266.175674] Moving to TARG_CONN_STATE_IN_LOGIN. [ 266.175712] i_buf: iqn.1993-08.org.debian:01:87ca34604, s_buf: Discovery, t_buf: (null) [ 266.184774] Got key: InitiatorName=iqn.1993-08.org.debian:01:87ca34604 [ 266.184780] iSCSI Parameter updated to InitiatorName=iqn.1993-08.org.debian:01:87ca34604 [ 266.184781] Got key: InitiatorAlias=barret [ 266.184785] iSCSI Parameter updated to InitiatorAlias=barret [ 266.184787] Got key: SessionType=Discovery [ 266.184791] iSCSI Parameter updated to SessionType=Discovery [ 266.184792] Got key: HeaderDigest=None [ 266.184796] iSCSI Parameter updated to HeaderDigest=None [ 266.184797] Got key: DataDigest=None [ 266.184800] iSCSI Parameter updated to DataDigest=None [ 266.184802] Got key: DefaultTime2Wait=2 [ 266.184806] iSCSI Parameter updated to DefaultTime2Wait=2 [ 266.184807] Got key: DefaultTime2Retain=0 [ 266.184811] iSCSI Parameter updated to DefaultTime2Retain=0 [ 266.184812] Got key: IFMarker=No [ 266.184816] iSCSI Parameter updated to IFMarker=No [ 266.184817] Got key: OFMarker=No [ 266.184821] iSCSI Parameter updated to OFMarker=No [ 266.184822] Got key: ErrorRecoveryLevel=0 [ 266.184826] iSCSI Parameter updated to ErrorRecoveryLevel=0 [ 266.184827] Got key: MaxRecvDataSegmentLength=8192 [ 266.184831] iSCSI Parameter updated to MaxRecvDataSegmentLength=8192 [ 266.184833] Saving op->MaxRecvDataSegmentLength from original initiator received value: 8192 [ 266.184837] iSCSI Parameter updated to MaxRecvDataSegmentLength=262144 [ 266.184838] Updated MaxRecvDataSegmentLength to target MXDSL value: 262144 [ 266.184848] iSCSI Parameter updated to IFMarker=No [ 266.184850] Reset "IFMarker" to "No". [ 266.184853] iSCSI Parameter updated to OFMarker=No [ 266.184855] Reset "OFMarker" to "No". [ 266.184858] iSCSI Parameter updated to IFMarkInt=Irrelevant [ 266.184860] Reset "IFMarkInt" to "Irrelevant". [ 266.184868] iSCSI Parameter updated to OFMarkInt=Irrelevant [ 266.184871] Reset "OFMarkInt" to "Irrelevant". [ 266.184873] Sending key: HeaderDigest=None [ 266.184884] Sending key: DataDigest=None [ 266.184894] Sending key: TargetAlias=LIO Target [ 266.184905] Sending key: TargetPortalGroupTag=1 [ 266.184915] Sending key: MaxRecvDataSegmentLength=262144 [ 266.184925] Sending key: DefaultTime2Wait=2 [ 266.184935] Sending key: DefaultTime2Retain=0 [ 266.184945] Sending key: ErrorRecoveryLevel=0 [ 266.184956] Sending Login Response, Flags: 0x87, ITT: 0x00000000, ExpCmdSN; 0x00000001, MaxCmdSN: 0x00000001, StatSN: 0xbb3cab31, Length: 173 [ 266.184966] tx_desc ffff88045cf8f880 lkey mismatch, fixing [ 266.184968] isert_init_tx_hdrs: Setup tx_sg[0].addr: 0x45cf8f880 length: 76 lkey: 0x00002d00 [ 266.185335] isert_post_recv(): Posted 32 RX buffers [ 266.185344] Moving to TARG_CONN_STATE_LOGGED_IN. [ 266.185346] ------------------------------------------------------------------ [ 266.185348] HeaderDigest: None [ 266.185349] DataDigest: None [ 266.185352] MaxXmitDataSegmentLength: 262144 [ 266.185353] MaxRecvDataSegmentLength: 8192 [ 266.185355] IFMarker: No [ 266.185356] OFMarker: No [ 266.185358] ------------------------------------------------------------------ [ 266.185360] ------------------------------------------------------------------ [ 266.185363] InitiatorName: iqn.1993-08.org.debian:01:87ca34604 [ 266.185365] TargetAlias: LIO Target [ 266.185366] InitiatorAlias: barret [ 266.185368] TargetPortalGroupTag: 1 [ 266.185370] DefaultTime2Wait: 2 [ 266.185371] DefaultTime2Retain: 0 [ 266.185373] ErrorRecoveryLevel: 0 [ 266.185375] SessionType: Discovery [ 266.185376] ------------------------------------------------------------------ [ 266.185380] Calling ib_dma_unmap_single for tx_desc->dma_addr [ 266.185440] Moving to TARG_SESS_STATE_LOGGED_IN. [ 266.185442] iSCSI Login successful on CID: 0 from 10.100.0.5 to 10.100.0.1:3261,1 [ 266.185444] Incremented iSCSI Connection count to 1 from node: iqn.1993-08.org.debian:01:87ca34604 [ 266.185446] Established iSCSI session from node: iqn.1993-08.org.debian:01:87ca34604 [ 266.185447] Incremented number of active iSCSI sessions to 1 on iSCSI Target Portal Group: 1 [ 266.185469] Moving to TARG_CONN_STATE_FREE. [ 266.186410] ISER req_buf: Using rx_dma: 0x853200000, rx_buflen: 8268 [ 266.186413] iSCSI opcode: 0x04, ITT: 0x00000000, flags: 0x80 dlen: 16 [ 266.186415] ISER ISCSI_CTRL PDU [ 266.186433] Got Text Request: ITT: 0x00000000, CmdSN: 0x01000000, ExpStatSN: 0x32ab3cbb, Length: 16 [ 266.186441] Received CmdSN matches ExpCmdSN, incremented ExpCmdSN to: 0x00000002 [ 266.186455] iSERT: Decremented post_recv_buf_count: 31 [ 266.186461] isert_post_recv(): Posted 32 RX buffers [ 266.186467] tx_desc ffff880458ee8508 lkey mismatch, fixing [ 266.186482] text_length: from iscsit_build_sendtargets_response: 83 [ 266.193485] padding: 1 [ 266.196116] Set outgoing TEXT response hdr->dlength: 83 [ 266.201952] Updated MaxCmdSN to 0x00000002 [ 266.201958] Built Text Response: ITT: 0x00000000, StatSN: 0xbb3cab32, Length: 83, CID: 0 [ 266.201960] isert_init_tx_hdrs: Setup tx_sg[0].addr: 0x458ee8508 length: 76 lkey: 0x00002d00 [ 266.201962] Posting Text Response IB_WR_SEND >>>>>>>>>>>>>>>>>>>>>> [ 266.201974] isert_send_completion: Got ISER_IB_SEND [ 266.201977] Calling ib_dma_unmap_single for tx_desc->dma_addr [ 266.201984] Calling ib_dma_unmap_single for isert_cmd->sense_buf_dma [ 266.201986] Entering isert_put_cmd: ffff880458ee8000 [ 266.201992] Entering isert_release_cmd ffff880458ee8000 >>>>>>>>>>>>>>>. [ 266.203677] isert_cma_handler: event 10 status 0 conn ffff88045cf8f3f0 id ffff88045d1e4480 [ 266.203679] RDMA_CM_EVENT_DISCONNECTED: >>>>>>>>>>>>>> [ 266.203684] isert_disconnect_work(): >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- To unsubscribe from this list: send the line "unsubscribe target-devel" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
