On 4/15/19 12:35 PM, Yuval Shaia wrote:
On Thu, Apr 11, 2019 at 07:02:15PM +0200, Cornelia Huck wrote:
On Thu, 11 Apr 2019 14:01:54 +0300
Yuval Shaia <yuval.shaia@xxxxxxxxxx> wrote:
Data center backends use more and more RDMA or RoCE devices and more and
more software runs in virtualized environment.
There is a need for a standard to enable RDMA/RoCE on Virtual Machines.
Virtio is the optimal solution since is the de-facto para-virtualizaton
technology and also because the Virtio specification
allows Hardware Vendors to support Virtio protocol natively in order to
achieve bare metal performance.
This RFC is an effort to addresses challenges in defining the RDMA/RoCE
Virtio Specification and a look forward on possible implementation
techniques.
Open issues/Todo list:
List is huge, this is only start point of the project.
Anyway, here is one example of item in the list:
- Multi VirtQ: Every QP has two rings and every CQ has one. This means that
in order to support for example 32K QPs we will need 64K VirtQ. Not sure
that this is reasonable so one option is to have one for all and
multiplex the traffic on it. This is not good approach as by design it
introducing an optional starvation. Another approach would be multi
queues and round-robin (for example) between them.
Typically there will be a one-to-one mapping between QPs and CPUs (on
the guest). So while one would need to be prepared to support quite some
QPs, the expectation is that the actual number of QPs used will be
rather low.
In a similar vein, multiplexing QPs would be defeating the purpose, as
the overall idea was to have _independent_ QPs to enhance parallelism.
Expectations from this posting:
In general, any comment is welcome, starting from hey, drop this as it is a
very bad idea, to yeah, go ahead, we really want it.
Idea here is that since it is not a minor effort i first want to know if
there is some sort interest in the community for such device.
My first reaction is: Sounds sensible, but it would be good to have a
spec for this :)
You'll need a spec if you want this to go forward anyway, so at least a
sketch would be good to answer questions such as how many virtqueues
you use for which purpose, what is actually put on the virtqueues,
whether there are negotiable features, and what the expectations for
the device and the driver are. It also makes it easier to understand
how this is supposed to work in practice.
If folks agree that this sounds useful, the next step would be to
reserve an id for the device type.
Thanks for the tips, will sure do that, it is that first i wanted to make
sure there is a use case here.
Waiting for any feedback from the community.
I really do like the ides; in fact, it saved me from coding a similar
thing myself :-)
However, I'm still curious about the overall intent of this driver.
Where would the I/O be routed _to_ ?
It's nice that we have a virtualized driver, but this driver is
intended to do I/O (even if it doesn't _do_ any I/O ATM :-)
And this I/O needs to be send to (and possibly received from)
something.
So what exactly is this something?
An existing piece of HW on the host?
If so, wouldn't it be more efficient to use vfio, either by using SR-IOV
or by using virtio-mdev?
Another guest?
If so, how would we route the I/O from one guest to the other?
Shared memory? Implementing a full-blown RDMA switch in qemu?
Oh, and I would _love_ to have a discussion about this at KVM Forum.
Maybe I'll manage to whip up guest-to-guest RDMA connection using
ivshmem ... let's see.
Cheers,
Hannes
--
Dr. Hannes Reinecke Teamlead Storage & Networking
hare@xxxxxxx +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Felix Imendörffer, Mary Higgins, Sri Rasiah
HRB 21284 (AG Nürnberg)
_______________________________________________
Virtualization mailing list
Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx
https://lists.linuxfoundation.org/mailman/listinfo/virtualization
