On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote: > Hi, > I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem based > server with IOH 5520. 5520 supports VTD. > I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter > which I assigned through VT-D to FreeBSD 8.2 running > as guest os. The ixgbe driver detects the device and the driver > successfully configures the device. But the link > never comes up. It looks like link up/down interrupts are not > delivered. Then I checked kvm interrupt assignment and as expected > kvm could not make MSI-X entries for the VT-d guest. So no output from > "grep kvm /proc/interrupt". By enabling some debugs in the > qemu-kvm I figured out that the MSI-x updates are not received > properly. It does look like Linux updates MSI-X table in a batch > fashion > which qemu-kvm gets in one shot and every thing works fine in case of > linux. In case of FreeBSD PCIE updates come /MSI-X entry > which qemu-kvm can't make use. That's right, Linux and Windows both seem to setup the MSI-X table then enable it in one shot, so we only trigger the interrupt programming when the enable bit is set. We don't trigger changes on writes to the MSI-X table... not very accurate emulation of mask bits. > To overcome this I compiled latest intel ixgbe driver with MSI-X > diabled. This time MSI interrupt got allocated both in the guest > and in the qemu-kvm (host kernel). Still I could not get link UP. I > modified the FreeBSD driver to poll for Link in some local_timer > task. Link comes up but no traffic flows. The MAC statistics show > packets received but packets do not reach the guest. > DMAing of packet may be failing. I could not find out the reason. > The same happens with legacy interrupt allocated in the guest. I even > tried qemu-kvm prefer_msi=off. > > I think there are two problems1 > 1. Interrupt delivery either because of interrupt remapping failure or > all out interrupt allocation failure in qemu-kvm > 2. Packets not getting DMAed to the guest possibly because of some DMAR issue. > > Before doing VT-d I made sure connections are fine and the adapter > works fine in bare metal Linux. > I also tried VT-d of 82599 with Linux as guest (both 32 bits(PAE and > non-PAE) and amd64) and it just works magically. Unknowns: does the intel ixgbe driver work in FreeBSD with MSI-X disabled, does polling for link UP work. The way you describe MSI-X interrupt setup for FreeBSD is likely something we don't handle correctly. You might do a hack in device-assignment.c to call assigned_dev_update_msix from msix_mmio_writel any time we write to the vector control dword. We really have no support for either function mask or per vector mask right now. I doubt you're having DMAR issues or you'd likely be seeing errors in the host dmesg. > Is this a known issue I am hitting with non-Linux guest oses and VT-d > ? Appreciate any help in debugging this problem. > > -Shashidhar > > Linux kernel - 2.6.35 > kvm - 0.14.1 It'd be helpful to test on something newer too. Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
