Re: [PATCH v3 rdma-core] verbs: Introduce a new reg_mr API for virtual address space

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Thu, May 30, 2019 at 09:56:09PM +0300, Yuval Shaia wrote:
> On Thu, May 30, 2019 at 06:17:21PM +0000, Jason Gunthorpe wrote:
> > On Thu, May 30, 2019 at 05:34:53PM +0300, Yuval Shaia wrote:
> > > On Thu, May 30, 2019 at 12:37:18PM +0000, Michal Kalderon wrote:
> > > > > From: linux-rdma-owner@xxxxxxxxxxxxxxx <linux-rdma-
> > > > > owner@xxxxxxxxxxxxxxx> On Behalf Of Yuval Shaia
> > > > > 
> > > > > The virtual address that is registered is used as a base for any address passed
> > > > > later in post_recv and post_send operations.
> > > > > 
> > > > > On a virtualized environment this is not correct.
> > > > > 
> > > > > A guest cannot register its memory so hypervisor maps the guest physical
> > > > > address to a host virtual address and register it with the HW. Later on, at
> > > > > datapath phase, the guest fills the SGEs with addresses from its address
> > > > > space.
> > > > > Since HW cannot access guest virtual address space an extra translation is
> > > > > needed to map those addresses to be based on the host virtual address that
> > > > > was registered with the HW.
> > > > > This datapath interference affects performances.
> > > > > 
> > > > > To avoid this, a logical separation between the address that is registered and
> > > > > the address that is used as a offset at datapath phase is needed.
> > > > > This separation is already implemented in the lower layer part
> > > > > (ibv_cmd_reg_mr) but blocked at the API level.
> > > > > 
> > > > > Fix it by introducing a new API function which accepts an address from guest
> > > > > virtual address space as well, to be used as offset for later datapath
> > > > > operations.
> > > > > 
> > > > Could you give an example of how an app would use this new API? How will
> > > > It receive the new hca_va addresss ? 
> > > 
> > > In my use case an application is device emulation that runs in the context
> > > of a userspace process in the host.
> > > This (virtual) device receives from guest driver a dma address (in form of
> > > scatter-gather list) along with guest user-space virtual address. 
> > 
> > How do you handle the scatter-gather list?
> 
> Well, it is not exactly scatter-gather, lets think of it as an array of
> guest dma addresses, in mellanox terms it is mtt, in vmware it is page
> directory.
> So i guess your question is how to register list of scattered addresses,
> right?

Yes..

I've always thought we should have an API to do that.

Jason



[Index of Archives]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Photo]     [Yosemite News]     [Yosemite Photos]     [Linux Kernel]     [Linux SCSI]     [XFree86]

  Powered by Linux