Hi Alex, Thanks for your inputs. We are using Mellanox ConnectX-3 iSER SRIOV capable NICs. We provision these VF's into the VM. The VM connects to few SSD drives through iSER. For this performance test, if we expose the same SSDs through iSER out of VM to servers & run vdbench 4K read/write workloads we see this significant performance drop when using vfio. These VM's have 8 hyper-threads from Intel E5-2680 v3 server & 32GB RAM. The key observation is with vfio the cpu saturates much earlier & hence cannot allow us to scale IOPs. I will open a separate mail thread about this performance degradation using vfio with numbers. In the meantime if you can suggest how to look for performance issue or what logs you would prefer for VFIO debugging it will help in getting the needed info for you. Thanks. --Shyam On Thu, Jan 7, 2016 at 7:40 PM, Alex Williamson <alex.williamson@xxxxxxxxxx> wrote: > On Thu, 2016-01-07 at 15:48 +0530, Shyam wrote: >> Hi All, >> >> We are using Linux Kernel 3.18.19 for running KVM VM's with >> pci-assign'ed SRIOV VF interfaces. >> >> We understand that VFIO is the new recommended way, but unfortunately >> it reduces performance significantly on our IO workloads (upto the >> order of 40-50%) when compared to pci-passthrough. We run trusted >> VM's >> & expose services to the external world. Since we control the VM's, >> IOMMU security with VFIO is not exactly mandatory, but performance is >> much more important that we get with pci-assign. >> >> We observe a strange behaviour that has already been discussed in >> this >> forum which is upon a VM spawn it causes the following fault >> resulting >> in qemu-kvm crashing >> >> Jan 7 09:41:57 q6-s1 kernel: [90037.228477] intel_iommu_map: iommu >> width (48) is not sufficient for the mapped address >> (fffffffffe001000) >> Jan 7 09:41:57 q6-s1 kernel: [90037.308229] >> kvm_iommu_map_address:iommu failed to map pfn=95000 >> >> We observe that this problem happens only if guest linux running 3.5 >> kernel is spun up & this problem doesnt happen when running guest >> linux with 3.6 kernel (i.e. all guest with kernels like 3.2 etc up >> till 3.5 causes the above crash whereas any guest kernel >=3.6 doesnt >> cause this issue). >> >> So something changed between kernel 3.5 to 3.6 in the guest that >> doesnt expose this problem. We have two questions: >> 1 - we understand that VFIO suffered a similar problem & it was fixed >> with https://github.com/qemu/qemu/commit/d3a2fd9b29e43e202315d5e99399 >> b99622469c4a. >> Alex Williamson suggested that KVM driver needs an equivalent version >> of the fix. Can anybody suggest hints on where this fix should be >> made? >> 2 - Any insights on what changes in linux kernel between 3.5 to 3.6 >> on >> the guest that avoids this problem? >> >> Any helps/input greatly appreciated. Thanks! > > Legacy KVM device assignment is deprecated, so I'd suggest your efforts > are better spent reporting and trying to fix any performance difference > you're seeing between pci-assign and vfio-pci. I have a really hard > time believing there's anywhere close to a 40-50% difference. What's > the device? What's the workload? At some point you're likely to find > that pci-assign is no longer even present. Thanks, > > Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
