On Thu, Mar 07, 2019 at 01:41:10AM +0000, Parav Pandit wrote: > Hi Yuval, > > > -----Original Message----- > > From: Bart Van Assche > > Sent: Wednesday, March 6, 2019 2:39 PM > > To: Yuval Shaia ; Parav Pandit > > Cc: Ira Weiny ; Leon Romanovsky ; Dennis Dalessandro ; linux- > > rdma@xxxxxxxxxxxxxxx; Marcel Apfelbaum ; Kamal Heib > > Subject: Re: [EXPERIMENTAL v1 0/4] RDMA loopback device > > > > On Wed, 2019-03-06 at 22:14 +0200, Yuval Shaia wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Suggestion: To enhance 'loopback' performances, can you > > > > > > > > > consider using shared memory or any other IPC instead of > > > > > > > > > going thought the > > > > > > > > > > > > > > network stack? > > > > > > > > > > > > > > > > > > > > > > > > > Loopback driver in this patchset doesn't use network stack. > > > > > > > > It is just 2000 lines of wrapper to memcpy() to enables > > > > > > > > applications to use > > > > > > > > > > > > > > rdma. > > > > > > > > > > > > > > To have a dedicated driver just for the loopback will force > > > > > > > the user to do a smart select, i.e. to use lo device for local > > > > > > > traffic and rxe for non- > > > > > > > > > > local. > > > > > > No. when application is written using rdmacm, everything works > > > > > > based on > > > > > > > > > > the ip address. > > > > > > It will pick the right rdma device that matches this ip. > > > > > > It would be 'lo' when connections are on 127.0.0.1. > > > > > > When application such as MPI, will have to anyway specify the > > > > > > which rdma > > > > > > > > > > device they want to use in system. > > > > > > > > > > But what if one wants to stay at the verb level and not use rdmacm > > API? > > > > > > > > > > > > > Sure. He can stay at verb level where he anyway have to explicitly give > > the device name. > > > > > > And that's is exactly the problem! > > > > > > With qemu, the ibdev is given at the command-line of the virtual > > > machine so if two guests starts on the same host it is ok to give them > > > the lo device as backend but what will happen when one of the VMs will > > > migrate to another host? The traffic will break since the lo device cannot > > go outside. > > > > Hi Yuval, > > > > I think what you are describing falls outside the use cases Parav has in mind. > > I think that optimizing RDMA over loopback, even if that loopback only > > works inside a single VM, is useful. > > > > lo rdma device takes birth inside the VM, migrates as pure memory to other host, just like any lo netdev, and dies in VM. > There is no need to give lo device from outside to the guest VM. Already answered to Bart's email so do not want to repeat my reply here. I see your point just do not want to turn one use case to a generic use, i.e. the same 'pure memcpy enhancement' requirements applies to a broader scope than your use case, enhancing rxe will hit them both while having yet another sw device will cover only your use case. Yuval
