Jason, Both the rxe driver and the siw driver running the blktests srp test suite exhibit failures on my machine running the for-next branch. This has been true for months so I decided to try again to track it down. After a lot of tracing, it looks like the problem is that the built in cq handling in core/cq.c is failing to continue to process some completion queues. The traffic is between the srp driver and the srpt driver. The srpt driver uses cq = ib_cq_pool_get(..., IB_POLL_WORKQUEUE) and the srp driver uses cq = ib_alloc_cq(..., IB_POLL_SOFTIRQ) for receive cqs and cq = ib_alloc_cq(..., IB_POLL_DIRECT) for send cqs. AFAIK the poll workqueue and poll softirq cqs are working correctly but the poll direct cq sometimes loses the thread and just stops processing those cqs. The test cases sometimes recover after about a 2 second delay and start processing again and eventually fail after about a 10 second delay and cleanup and go home. The failures feel like a race or at least are timing sensitive. If you run the test suite several times various test cases will sometimes succeed and sometimes fail. But they always fail in the same way. Looking at the mlxn drivers for inspiration, I don't see anything specific about IB_POLL_DIRECT except that they have a private version of send_queue_drain which also calls a cqe drain function which calls ib_process_cq_direct() in a loop until the cq is drained. But this is only during qp tear down. (No other verbs driver does this but as far as I know no other driver is passing blktests.) This is only done for IB_POLL_DIRECT, so I wonder, is this required to use that correctly? I am still figuring out how IB_POLL_DIRECT works. It doesn't allow the driver to call cq->comp_handler so I don't know how it figures out when there are new wcs to process. Any ideas would be really helpful. Bob
