Sagi, I’m seeing IOs getting timed out. In the target side (in dmesg) I see many the following messages (when the SQ post fails): [419075.424558] isert: isert_rdma_rw_ctx_post: Cmd: 000000008ee38b5c failed to post RDMA res [419075.425768] isert: isert_post_response: ib_post_send failed with -12 On the initiator side I’m seeing errors from ‘maim’ application: 10/28/18 13:03:10 Thread:8 Error:17 - An I/O has been pending in excess of monitor watch time! (LBA:0x015400 reqSize:524288 retSize:0) 10/28/18 13:03:12 Sampler: Error:20 - I/O HALT! No I/O completion sampled in monitor watch time! Thanks, Amit. From: Sagi Grimberg <sagi@xxxxxxxxxxxxxxxxx> Hi Amit, The isert code indeed might attempt to post over the QP limits and expect it to fail if there is not enough room, in which case the scsi target queue_full logic should step in... So while this can happen, Do you see user I/O impacted? or simply some debug info popping up? Sagi. On Wed, 24 Oct 2018 at 02:09, Radzi, Amit <mailto:Amit.Radzi@xxxxxxxxxx> wrote: Sagi, We encountered an issue where the isert code overflows the RDMA QP SQ. On QP creation the isert initializes 'max_send_wr' parameter according to the number of expected commands: attr.cap.max_send_wr = ISERT_QP_MAX_REQ_DTOS + 1; The issues seem to be that for some commands the number of WQEs posted to the SQ is more than just one. The RDMA RW layer used to post the SQ WQEs may create multiple WQEs depending on the context and MR type and the number of SCSI SGEs. In the scenario we are running the function 'rdma_rw_init_map_wrs' creates the number of WQEs by dividing the number of SGEs by the number of SGEs supported by the device: u32 max_sge = dir == DMA_TO_DEVICE ? qp->max_write_sge : qp->max_read_sge; ctx->nr_ops = DIV_ROUND_UP(sg_cnt, max_sge); Could you please have a look? Thanks, Amit.
