Jason,
Fundamentally you must drive SQ flow control via CQ completions. For instance a ULP cannot disable all CQ notifications and keep stuffing things into the SQ. An alternative way to state this: A ULP cannot use activity on the RQ to infer that there is space in the SQ. Only CQ completions can be used to prove there is more available SQ space. Do not post to the SQ until a CQ has been polled proving available space.
The recv queue is out of the ball game here... We just selectively signal send completions only to reduce some interrupts and completion processing.
Ultimately you need a minimum of one CQ notification for every SQ depth post and the ULP must not post to the SQ once it fills until it sees the CQ notification. That usually drives the rule of thumb to notify every 1/2 depth, however any SQWE posting failures indicate a ULP bug..
Well, usually, but I'm not convinced this is the case. For each I/O we post up to 2 work requests, 1 for memory registration and 1 for sending an I/O request (and 1 for local invalidation if the target doesn't do it for us, but that is not the case here). So if our queue depth is X, we size our completion queue to be X*3, and we need to make sure we signal every (X*3)/2. What I think Marta is seeing, is that no matter how long we size our send-queue (in other words no matter how high X is) we're not able to get away with suppress send completion signaling for more than ~100 posts. Please correct me if I'm wrong Marta? -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
