On Wed, Sep 11, 2024 at 12:24:07PM -0500, Doug Miller wrote: > On 9/11/2024 11:12 AM, Mathieu Poirier wrote: > > On Tue, 10 Sept 2024 at 09:43, Doug Miller > > <doug.miller@xxxxxxxxxxxxxxxxxxxx> wrote: > > > On 9/10/2024 10:13 AM, Mathieu Poirier wrote: > > > > On Tue, Sep 10, 2024 at 08:12:07AM -0500, Doug Miller wrote: > > > > > On 9/3/2024 10:52 AM, Doug Miller wrote: > > > > > > I am trying to learn how to create an RPMSG-over-VIRTIO device > > > > > > (service) in order to perform communication between a host driver and > > > > > > a guest driver. The RPMSG-over-VIRTIO driver (client) side is fairly > > > > > > well documented and there is a good example (starting point, at least) > > > > > > in samples/rpmsg/rpmsg_client_sample.c. > > > > > > > > > > > > I see that I can create an endpoint (struct rpmsg_endpoint) using > > > > > > rpmsg_create_ept(), and from there I can use rpmsg_send() et al. and > > > > > > the rpmsg_rx_cb_t cb to perform the communications. However, this > > > > > > requires a struct rpmsg_device and it is not clear just how to get one > > > > > > that is suitable for this purpose. > > > > > > > > > > > > It appears that one or both of rpmsg_create_channel() and > > > > > > rpmsg_register_device() are needed in order to obtain a device for the > > > > > > specific host-guest communications channel. At some point, a "root" > > > > > > device is needed that will use virtio (VIRTIO_ID_RPMSG) such that new > > > > > > subdevices can be created for each host-guest pair. > > > > > > > > > > > > In addition, building a kernel with CONFIG_RPMSG, CONFIG_RPMSG_VIRTIO, > > > > > > and CONFIG_RPMSG_NS set, and doing a modprobe virtio_rpmsg_bus, seems > > > > > > to get things setup but that does not result in creation of any "root" > > > > > > rpmsg-over-virtio device. Presumably, any such device would have to be > > > > > > setup to use a specific range of addresses and also be tied to > > > > > > virtio_rpmsg_bus to ensure that virtio is used. > > > > > > > > > > > > It is also not clear if/how register_rpmsg_driver() will be required > > > > > > on the rpmsg driver side, even though the sample code does not use it. > > > > > > > > > > > > So, first questions are: > > > > > > > > > > > > * Am I looking at the correct interfaces in order to create the host > > > > > > rpmsg device side? > > > > > > * What needs to be done to get a "root" rpmsg-over-virtio device > > > > > > created (if required)? > > > > > > * How is a rpmsg-over-virtio device created for each host-guest driver > > > > > > pair, for use with rpmsg_create_ept()? > > > > > > * Does the guest side (rpmsg driver) require any special handling to > > > > > > plug-in to the host driver (rpmsg device) side? Aside from using the > > > > > > correct addresses to match device side. > > > > > It looks to me as though the virtio_rpmsg_bus module can create a > > > > > "rpmsg_ctl" device, which could be used to create channels from which > > > > > endpoints could be created. However, when I load the virtio_rpmsg_bus, > > > > > rpmsg_ns, and rpmsg_core modules there is no "rpmsg_ctl" device created > > > > > (this is running in the host OS, before any VMs are created/run). > > > > > > > > > At this time the modules stated above are all used when a main processor is > > > > controlling a remote processor, i.e via the remoteproc subsystem. I do not know > > > > of an implementation where VIRTIO_ID_RPMSG is used in the context of a > > > > host/guest scenario. As such you will find yourself in uncharted territory. > > > > > > > > At some point there were discussion via the OpenAMP body to standardize the > > > > remoteproc's subsystem establishment of virtqueues to conform to a host/guest > > > > scenario but was abandonned. That would have been a step in the right direction > > > > for what you are trying to do. > > > I was looking at some existing rpmsg code, at it appeared to me that > > > some adapters, like the "qcom", are creating an rpmsg device that > > > provides specialized methods for talking to the remote processor(s). I > > > have assumed this is because that hardware does not allow for running > > > something remotely that can utilize the virtio queues directly, and so > > > these rpmsg devices provide code to do the communication with their > > > hardware. What's not clear is whether these devices are using > > > rpmsg-over-virtio or if they are creating their own rpmsg facility (and > > > whether they even support guest-host communication). > > > > > The QC implementation is different and does not use virtio - there is > > a special HW interface between the main and the remote processors. > > That configuration is valid since RPMSG can be implemented over > > anything. > > > > > What I'm also wondering is what needs to be done differently for virtio > > > when communicating guest-host vs local CPU to remote processor. I was > > From a kernel/guest perspective, not much should be needed. That said > > the VMM will need to be supplemented with extra configuration > > capabilities to instantiate the virtio-rpmsg device. But that is just > > off the top of my head without seriously looking at the use case. > > From a virtio-bus perspective, there might be an issue if a platform > > is using remote processors _and_ also instantiating VMs that > > configures a virtio-rpmsg device. Again, that is just off the top of > > my head but needs to be taken into account. > I am new to rpmsg and virtio, and so my understanding of internals is > still very limited. Is there someone I can work with to determine what > needs to be done here? I am guessing that virtio either automatically > adapts to guest-host or rproc-host - in which case no changes may be > required - or else it requires a different setup and rpmsg will need to > be extended to allow for that. If there are changes to rpmsg required, > we'll want to get those submitted as soon as possible. One complication > for submitting our driver changes is that it is part of a much larger > effort to support new hardware, and it may not be possible to submit > them together with rpmsg changes. The virtio part won't be a problem. In your case what is missing is the glue that will setup the virtqueues and install the RPMSG protocol on top of them. The 'glue' is the new virtio-rpmsg device that needs to be created. That part includes the creation of a new virtio device by the VMM and a kernel driver that can be called from the virtio_bus once it has been discovered. Everything in the virtio and RPMSG subsystems are aleady tailored to support all this, so no changes should be needed. As for the VMM, I suggest to start with kvmtool. Lastly, none of this requires "real" hardware or your specific hardware - it can all be done from QEMU. > > > > > hoping that RPMSG-over-VIRTIO would be easily adapted to what we need. > > > If we have to create a new virtio device (that is nearly identical to > > > rpmsg), that is going to push-out SR-IOV support a great deal, plus > > > requiring cloning of a lot of existing code for a new purpose. > > Duplication of code would not be a viable way forward. > > Reusing/enhancing/fixing what is currently available is definitely a > > better option. > > > > > Our only other alternative is to do something to allow guest-host > > > communication to use the fabric loopback, which is not at all desirable > > > and has many issues of its own. > > > > > > > > Is this the correct way to use RPMSG-over-VIRTIO? If so, what actions > > > > > need to be taken to cause a "rpmsg_ctl" device to be created? What > > > > > method would be used (in a kernel driver) to get a pointer to the > > > > > "rpmsg_ctl" device, for use with rpmsg_create_channel()? > > > > > > > > > > > Thanks, > > > > > > Doug > > > > > > > > > > > External recipient > > > > > > External recipient > > > External recipient
