On Wed, Feb 27, 2013 at 03:50:54AM -0800, Nikhil Rao wrote: > > We are implementing a driver for a PCIe card that runs Linux. This card > needs "virtual" network/disk/console devices, so we have reused the > virtio devices on on the card and provided a host backend that interacts > with the virtio devices through the card's driver. > > this approach is very much like what was proposed on this thread > http://permalink.gmane.org/gmane.linux.ports.sh.devel/10379 > > We will posting the driver soon, so perhaps I am jumping the gun with my > question below about replacing our backend with vhost. > > It is possible for vhost (along with vhost-net in the case of > virtio-net) to serve as the backend. The copy between virtio buffers and > skbs happens in the tun/tap driver which means tun/tap may need to use a > HW DMA engine (the card has one) for copy across the bus to get close to > the full PCIe bandwidth. > > tun/tap was probably never designed for this use case, but reusing vhost > does simplify our backend since it is now only involved in setup and > potentially has a performance/memory footprint advantage due to avoiding > context switches/intermediate buffer copy and this idea can be > generalized to other cards as well. > > Comments/suggestions ? > > Thanks, > Nikhil > > _______________________________________________ > Virtualization mailing list > Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx > https://lists.linuxfoundation.org/mailman/listinfo/virtualization Hi Nikhil, I don't have any code to offer, but may be able to provide some suggestions. I work on a system which has a single (x86) host computer, and many PowerPC data processing boards, which are connected via PCI. This sounds similar to your hardware. Our system was developed before vhost existed. I built a (fairly dumb) network driver that just transfers packets over PCI using the PowerPC DMA controller. It works, however I think a more generic virtio solution will work better. A virtio solution will also allow other types of devices besides a network interface. I have done some studying of rproc/rpmsg/vhost/vringh, and may have some suggestions about those pieces of kernel functionality. A HW DMA engine is absolutely needed to get good performance over the PCI bus. I don't have experience with PCIe. You may want to investigate rproc/rpmsg to help do virtio device discovery. When dealing with virtio, it may be helpful to think of your PCIe card as the "host". In virtio nomenclature, the "host" is in charge of copying data. Your HW DMA engine needs to be controlled by the "host". Your main computer (the computer the PCIe card plugs into) will be a virtio "guest" and will run the virtio-net/virtio-console/etc. drivers. Several vendors have contacted me privately to ask for the code for my (dumb) network-over-PCI driver. A generic solution to this problem will definitely find a userbase. I look forward to porting the code to run on my PowerPC PCI boards when it becomes available. I am able to help review code as well. Good luck! Ira _______________________________________________ Virtualization mailing list Virtualization@xxxxxxxxxxxxxxxxxxxxxxxxxx https://lists.linuxfoundation.org/mailman/listinfo/virtualization
