Started with Linus's tree, applied the change requested by Sagi, built the kernel, rebooted and started the tests. Linux ibclient 4.10.0-rc8.sagi+ #1 SMP Wed Feb 15 11:09:44 EST 2017 x86_64 x86_64 x86_64 GNU/Linux Very quickly get to this [ 180.990285] mlx5_0:dump_cqe:262:(pid 0): dump error cqe [ 181.016899] 00000000 00000000 00000000 00000000 [ 181.040949] 00000000 00000000 00000000 00000000 [ 181.066960] 00000000 00000000 00000000 00000000 [ 181.092030] 00000000 0f007806 2500002a bf1913d0 [ 181.117254] scsi host2: ib_srp: failed FAST REG status memory management operation error (6) for CQE ffff880bdbe88778 [ 196.288933] fast_io_fail_tmo expired for SRP port-2:1 / host2. [ 197.090886] scsi host2: ib_srp: reconnect succeeded [ 197.127628] scsi host2: ib_srp: failed RECV status WR flushed (5) for CQE ffff8817f09b6f30 So does not help. I think my and Barts suggestion to revert for now is the best way forward. I have already tested this in-depth from Bart's tree and its been sent to Doug as V2 of Bart'recent 8 patch series.
Yea, probably this is the best way forward. Bart, I think the change I suggested is still needed regardless, do you agree? Max, Leon, is it possible that the max number of klms pr mr is less than what reported in device capabilities for page_list_len? If so, this means that either: 1. mlx5 needs to expose the minimum between pages and sg elems (sucks) 2. we need yet another capability for SG_GAPS (sucks^2 because the whole point was to make it transparent to the user) 3. mlx5 does not support SG_GAPS (sucks^3 because we now have something thats not supported by any device).