> On Nov 15, 2018, at 2:10 AM, Syed Syed <SYEDS@xxxxxxxxxx> wrote: > > Hi Hi Syed, > While implementing user verbs, especially for rdma send operation, I wanted to check with the list to confirm our understanding. > The typical flow is the user space application allocates memory(VA), and then it registers memory through reg_mr verb. Correct. > For SEND > operation only L_KEY is needed, and R_KEY is not needed. Correct, rkey is used for remote operations such as RDMA WRITE and READ. In this case, during registration you need to specify the relevant access flags in reg mr.. > As usual the kernel pins this memory via ib_umem_get. Later the application > shall call ibv_post_send with the VA (if its server), if its client it shall call ibv_post_recv with the VA. The responser side should call the ibv_post_recv before the requestor calls the ibv_post_send.. > The verbs implementation, shall > program hw wqe via mmap'ed registers. Later the application > shall invoke doorbell via mmap'ed hw registers. The library rings the doorbell and not the application.. Also I guess it depends on HW implementation:) > The underlying assumption being the HW shall use its own implementation(mtt or other) to figure the right PA for this VA and do the > send or receive. Correct.. Again depends on HW.. it should maintain VA->DMA address mapping in order to do DMA .. > All this scheme happens with complete kernel bypass except for memory registration step. Control path goes through kernel and it includes create, modify, destroy of objects and memory registration. Data path, which is post send, recv and poll CQ is done in kernel bypass.. > > But for kernel space QP1(gsi), the kernel is posing wr's for send/receive with dma_addr/PA, and as such the memory is not registered > in this case. Correct.. > > Thanks > -syed > > This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.
