Hey folks, It's been a long time since the 3rd, in the meantime, some RXE regressions have been fixed by comminity. So It'd like to post my 4th version. feedbacks are very welcome :). Thanks. These patches are going to implement a *NEW* RDMA opcode "RDMA FLUSH". In IB SPEC 1.5[1], 2 new opcodes, ATOMIC WRITE and RDMA FLUSH were added in the MEMORY PLACEMENT EXTENSIONS section. This patchset makes SoftRoCE support new RDMA FLUSH on RC service. You can verify the patchset by building and running the rdma_flush example[2]. server: $ ./rdma_flush_server -s [server_address] -p [port_number] client: $ ./rdma_flush_client -s [server_address] -p [port_number] Corresponding pyverbs and tests(tests.test_qpex.QpExTestCase.test_qp_ex_rc_rdma_flush) are also added to rdma-core This patches do: - Make memory region support FLUSH access flags - Make HCA/device support FLUSH capabilities(placement type attributes). - Implement new packet for FLUSH request. [1]: https://www.infinibandta.org/wp-content/uploads/2021/08/IBTA-Overview-of-IBTA-Volume-1-Release-1.5-and-MPE-2021-08-17-Secure.pptx [2]: https://github.com/zhijianli88/rdma-core/tree/rdma-flush CC: Xiao Yang <yangx.jy@xxxxxxxxxxx> CC: "Gotou, Yasunori" <y-goto@xxxxxxxxxxx> CC: Jason Gunthorpe <jgg@xxxxxxxx> CC: Zhu Yanjun <zyjzyj2000@xxxxxxxxx CC: Leon Romanovsky <leon@xxxxxxxxxx> CC: Bob Pearson <rpearsonhpe@xxxxxxxxx> CC: Mark Bloch <mbloch@xxxxxxxxxx> CC: Wenpeng Liang <liangwenpeng@xxxxxxxxxx> CC: Aharon Landau <aharonl@xxxxxxxxxx> CC: Tom Talpey <tom@xxxxxxxxxx> CC: "Gromadzki, Tomasz" <tomasz.gromadzki@xxxxxxxxx> CC: Dan Williams <dan.j.williams@xxxxxxxxx> CC: linux-rdma@xxxxxxxxxxxxxxx CC: linux-kernel@xxxxxxxxxxxxxxx Can also access the kernel source in: https://github.com/zhijianli88/linux/tree/rdma-flush Changes log V4: - rework responder process - rebase to v5.19+ - remove [7/7]: RDMA/rxe: Add RD FLUSH service support since RD is not really supported V3: - Just rebase and commit log and comment updates - delete patch-1: "RDMA: mr: Introduce is_pmem", which will be combined into "Allow registering persistent flag for pmem MR only" - delete patch-7 V2: RDMA: mr: Introduce is_pmem check 1st byte to avoid crossing page boundary new scheme to check is_pmem # Dan RDMA: Allow registering MR with flush access flags combine with [03/10] RDMA/rxe: Allow registering FLUSH flags for supported device only to this patch # Jason split RDMA_FLUSH to 2 capabilities RDMA/rxe: Allow registering persistent flag for pmem MR only update commit message, get rid of confusing ib_check_flush_access_flags() # Tom RDMA/rxe: Implement RC RDMA FLUSH service in requester side extend flush to include length field. # Tom and Tomasz RDMA/rxe: Implement flush execution in responder side adjust start for WHOLE MR level # Tom don't support DMA mr for flush # Tom check flush return value RDMA/rxe: Enable RDMA FLUSH capability for rxe device adjust patch's order. move it here from [04/10] Li Zhijian (6): RDMA: Allow registering MR with flush access flags RDMA/rxe: Allow registering persistent flag for pmem MR only RDMA/rxe: Implement RC RDMA FLUSH service in requester side RDMA/rxe: Implement flush execution in responder side RDMA/rxe: Implement flush completion RDMA/rxe: Enable RDMA FLUSH capability for rxe device drivers/infiniband/sw/rxe/rxe_comp.c | 4 +- drivers/infiniband/sw/rxe/rxe_hdr.h | 48 ++++++++ drivers/infiniband/sw/rxe/rxe_loc.h | 2 + drivers/infiniband/sw/rxe/rxe_mr.c | 23 +++- drivers/infiniband/sw/rxe/rxe_opcode.c | 21 ++++ drivers/infiniband/sw/rxe/rxe_opcode.h | 4 + drivers/infiniband/sw/rxe/rxe_param.h | 4 +- drivers/infiniband/sw/rxe/rxe_req.c | 15 ++- drivers/infiniband/sw/rxe/rxe_resp.c | 149 +++++++++++++++++++++++- include/rdma/ib_pack.h | 2 + include/rdma/ib_verbs.h | 19 ++- include/uapi/rdma/ib_user_ioctl_verbs.h | 2 + include/uapi/rdma/ib_user_verbs.h | 14 +++ include/uapi/rdma/rdma_user_rxe.h | 7 ++ 14 files changed, 302 insertions(+), 12 deletions(-) -- 2.31.1
