Hi Yuval, +AD4- -----Original Message----- +AD4- From: Bart Van Assche +AD4- Sent: Wednesday, March 6, 2019 2:39 PM +AD4- To: Yuval Shaia +ADs- Parav Pandit +AD4- Cc: Ira Weiny +ADs- Leon Romanovsky +ADs- Dennis Dalessandro +ADs- linux- +AD4- rdma+AEA-vger.kernel.org+ADs- Marcel Apfelbaum +ADs- Kamal Heib +AD4- Subject: Re: +AFs-EXPERIMENTAL v1 0/4+AF0- RDMA loopback device +AD4- +AD4- On Wed, 2019-03-06 at 22:14 +-0200, Yuval Shaia wrote: +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- Suggestion: To enhance 'loopback' performances, can you +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- consider using shared memory or any other IPC instead of +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- going thought the +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- network stack? +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- Loopback driver in this patchset doesn't use network stack. +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- It is just 2000 lines of wrapper to memcpy() to enables +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- applications to use +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- rdma. +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- To have a dedicated driver just for the loopback will force +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- the user to do a smart select, i.e. to use lo device for local +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- traffic and rxe for non- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- local. +AD4- +AD4- +AD4- +AD4- +AD4- No. when application is written using rdmacm, everything works +AD4- +AD4- +AD4- +AD4- +AD4- based on +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- the ip address. +AD4- +AD4- +AD4- +AD4- +AD4- It will pick the right rdma device that matches this ip. +AD4- +AD4- +AD4- +AD4- +AD4- It would be 'lo' when connections are on 127.0.0.1. +AD4- +AD4- +AD4- +AD4- +AD4- When application such as MPI, will have to anyway specify the +AD4- +AD4- +AD4- +AD4- +AD4- which rdma +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- device they want to use in system. +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- But what if one wants to stay at the verb level and not use rdmacm +AD4- API? +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- +AD4- Sure. He can stay at verb level where he anyway have to explicitly give +AD4- the device name. +AD4- +AD4- +AD4- +AD4- And that's is exactly the problem+ACE- +AD4- +AD4- +AD4- +AD4- With qemu, the ibdev is given at the command-line of the virtual +AD4- +AD4- machine so if two guests starts on the same host it is ok to give them +AD4- +AD4- the lo device as backend but what will happen when one of the VMs will +AD4- +AD4- migrate to another host? The traffic will break since the lo device cannot +AD4- go outside. +AD4- +AD4- Hi Yuval, +AD4- +AD4- I think what you are describing falls outside the use cases Parav has in mind. +AD4- I think that optimizing RDMA over loopback, even if that loopback only +AD4- works inside a single VM, is useful. +AD4- lo rdma device takes birth inside the VM, migrates as pure memory to other host, just like any lo netdev, and dies in VM. There is no need to give lo device from outside to the guest VM.
