On Mon, 26 Aug 2024 at 11:22, Doug Miller <doug.miller@xxxxxxxxxxxxxxxxxxxx> wrote: > > On 8/26/2024 11:50 AM, Mathieu Poirier wrote: > > Apologies for the late reply - this got lost in the vacation email backlog. > > > > On Mon, 26 Aug 2024 at 10:27, Dennis Dalessandro > > <dennis.dalessandro@xxxxxxxxxxxxxxxxxxxx> wrote: > >> On 7/31/24 4:02 PM, Doug Miller wrote: > >>> I am working on SR-IOV support for a new adapter which has shared > >>> resources between the PF and VFs and requires an out-of-band (outside > > It would have been a good idea to let people know what "PF" and "VF" > > means to avoid confusion. > "PF" refers to the Physical Function of the PCI adapter - that which > exists always, regardless of whether SR-IOV is active. The "VF" refers > to the virtual function(s) that are created when SR-IOV is enabled and > configured. Typically, the VFs and the PF are assigned to different OS > instances running in different VMs. So, the OS that owns the PF needs to > be able to handle resource requests from the OSes that own the VFs (and > also send notifications). Thank you for the clarification. > > > >>> the adapter) communication mechanism to manage those resources. I have > >>> been looking at RPMSG as a mechanism to communicate between the driver > >>> on a guest (VM) and the driver on the host OS (which "owns" the > >>> resources). It appears to me that virtio is intended for communication > >>> between guests and host, and RPMSG over virtio is what I want to use. > >>> > > Virtio is definitely the standard way to convey information between a > > host and a guest. You can specify as many virtqueues as needed > > (in-band and out-of-band) and it is widely supported. What > > information is conveyed by the virtqueues and how it gets conveyed is > > entirely up to the use case. Have a look at the specification of > > existing virtio drivers to get a better idea [1]. If the driver you > > are working with hasn't been standardised, I highly encourage you to > > submit a draft for it. If it has then add to the current > > specification. > > > > All that said, you could use RPMSG as the protocol that runs on top of > > the virtqueues - that should be fairly easy to do. > I had initially started looking at using virtio directly, but it looked > like I was going to have to get a new device ID defined upstream and it > would be a significant effort compared to using an existing facility. I > then saw device ID VIRTIO_ID_RPMSG, which appears to be exactly what > we'd have to create if we were defining a new virtio device for what we > need. However, the problem has been understanding how to write code to > provide the rpmsg "device" side. There does not appear to be any > documentation and there is no example code to follow. It seems that the > device side is typically contained in a GPU or accelerator, which was > not written for a Linux kernel. So I have many questions on how (and > when) to use the interfaces (rpmsg_register_device, > rpmsg_create_channel, rpmsg_create_ept, rpmsg_find_device, ...). VIRTIO_ID_RPMSG is a special case - it was defined to establish a communication channel between a main processor (typically a cortex-A) and a remote processor, something like a M4 or an R5F. As such it is typically used in conjunction with the "remoteproc" subsystem. The device side you are looking for is part of the openAMP library [1]. I am not aware of an implementation of a virtio device that would use VIRTIO_ID_RPMSG in a MMIO area or a PCI config space to instantiate a generic message passing interface. [1]. https://github.com/OpenAMP/open-amp > > > > Thanks, > > Mathieu > > > > [1]. https://docs.oasis-open.org/virtio/virtio/v1.2/csd01/virtio-v1.2-csd01.html > > > >>> Can anyone confirm that RPMSG is capable of doing what we need? If so, > >>> I'll need some help figuring out how to use that from kernel device > >>> drivers (I've not been able to find any examples of doing the > >>> service/device side). If not, is there some other facility that is > >>> better suited? > >> Hi Bjorn and Mathieu, any advice here for Doug? Adding linux-rdma folks as that > >> is where this will eventually target. > >> > >> -Denny > >> > > External recipient
