> On Nov 24, 2015, at 9:44 AM, Sagi Grimberg <sagig@xxxxxxxxxxxxxxxxxx> wrote: > > Hey Chuck, > >> >>> It is painful, too painful. The entire value proposition of RDMA is >>> low-latency and waiting for the extra HW round-trip for a local >>> invalidation to complete is unacceptable, moreover it adds a huge loads >>> of extra interrupts and cache-line pollutions. >> >> The killer is the extra context switches, I’ve found. > > That too... > >> I’ve noticed only a marginal loss of performance on modern >> hardware. > > Would you mind sharing your observations? I’m testing with CX-3 Pro on FDR. NFS READ and WRITE round trip latency, which includes the cost of registration and now invalidation, is not noticeably longer. dbench and fio results are marginally slower (in the neighborhood of 5%). For NFS, the cost of invalidation is probably not significant compared to other bottlenecks in our stack (lock contention and scheduling overhead are likely the largest contributors). Notice that xprtrdma chains together all the LOCAL_INV WRs for an RPC, and only signals the final one. Before, every LOCAL_INV WR was signaled. So this patch actually reduces the send completion rate. The main benefit for NFS of waiting for invalidation to complete is better send queue accounting. Even without the data integrity issue, we have to ensure the WQEs consumed by invalidation requests are released before dispatching another RPC. Otherwise the send queue can be overrun. -- Chuck Lever -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
