> -----Original Message----- > From: Michael S. Tsirkin [mailto:mst@xxxxxxxxxx] > Sent: Monday, 11 April, 2016 3:28 PM > To: Markus Armbruster > Cc: Eli Britstein; qemu-devel@xxxxxxxxxx; kvm@xxxxxxxxxxxxxxx > Subject: Re: IVSHMEM device performance > > On Mon, Apr 11, 2016 at 10:56:54AM +0200, Markus Armbruster wrote: > > Cc: qemu-devel > > > > Eli Britstein <eli.britstein@xxxxxxxxxxxxxxxx> writes: > > > > > Hi > > > > > > In a VM, I add a IVSHMEM device, on which the MBUFS mempool > resides, and also rings I create (I run a DPDK application in the VM). > > > I saw there is a performance penalty if I use such device, instead of > hugepages (the VM's hugepages). My VM's memory is *NOT* backed with > host's hugepages. > > > The memory behind the IVSHMEM device is a host hugepage (I use a > patched version of QEMU, as provided by Intel). > > > I thought maybe the reason is that this memory is seen by the VM as a > mapped PCI memory region, so it is not cached, but I am not sure. > > > So, my direction was to change the kernel (in the VM) so it will consider > this memory as a regular memory (and thus cached), instead of a PCI > memory region. > > > However, I am not sure my direction is correct, and even if so, I am not > sure how/where to change the kernel (my starting point was mm/mmap.c, > but I'm not sure it's the correct place to start). > > > > > > Any suggestion is welcomed. > > > Thanks, > > > Eli. > > A cleaner way is just to use virtio, keeping everything in VM's memory, with > access either by data copies in hypervisor, or directly using vhost-user. > For example, with vhost-pci: https://wiki.opnfv.org/vm2vm_mst there has > been recent work on this, see slides 12-14 in > http://schd.ws/hosted_files/ons2016/36/Nakajima_and_Ergin_PreSwitch_fi > nal.pdf > > This is very much work in progress, but if you are interested you should > probably get in touch with Nakajima et al. [Eli Britstein] This is indeed very interesting and I will further look into it. However, if I'm not wrong, this requires some support from the host which I would like to avoid. My requirement from the host is only to provide an IVSHMEM device for several VMs, and my applications are running on VMs only. So, I think that vhost-pci is not applicable in my case. Am I wrong? Can you think of a reason why accessing that PCI memory mapped memory (which is really the host's hugepage) is more expensive than accessing the VM's hugepages (even though they are not really the host's hugepage)? Do you think my suspicion that as a PCI mapped memory it doesn't use cache is correct? If so, do you think I can change it (either in some configuration or changing the VM's kernel)? Any other direction? Thanks, Eli > > -- > MST ------------------------------------------------------------------------------------------------------------------------------------------------- This email and any files transmitted and/or attachments with it are confidential and proprietary information of Toga Networks Ltd., and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information of Toga Networks Ltd., and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. ------------------------------------------------------------------------------------------------------------------------------------------------ -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
