On Fri, 2012-01-13 at 12:25 +0530, Shashidhar Patil wrote: > Hi Alex, > Thanks for your help. Some answers and further investigation details > inline. > > On Wed, Jan 4, 2012 at 8:51 AM, Alex Williamson > <alex.williamson@xxxxxxxxxx>wrote: > > > On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote: > > > Hi, > > > I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem based > > > server with IOH 5520. 5520 supports VTD. > > > I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter > > > which I assigned through VT-D to FreeBSD 8.2 running > > > as guest os. The ixgbe driver detects the device and the driver > > > successfully configures the device. But the link > > > never comes up. It looks like link up/down interrupts are not > > > delivered. Then I checked kvm interrupt assignment and as expected > > > kvm could not make MSI-X entries for the VT-d guest. So no output from > > > "grep kvm /proc/interrupt". By enabling some debugs in the > > > qemu-kvm I figured out that the MSI-x updates are not received > > > properly. It does look like Linux updates MSI-X table in a batch > > > fashion > > > which qemu-kvm gets in one shot and every thing works fine in case of > > > linux. In case of FreeBSD PCIE updates come /MSI-X entry > > > which qemu-kvm can't make use. > > > > That's right, Linux and Windows both seem to setup the MSI-X table then > > enable it in one shot, so we only trigger the interrupt programming when > > the enable bit is set. We don't trigger changes on writes to the MSI-X > > table... not very accurate emulation of mask bits. > > > > > To overcome this I compiled latest intel ixgbe driver with MSI-X > > > diabled. This time MSI interrupt got allocated both in the guest > > > and in the qemu-kvm (host kernel). Still I could not get link UP. I > > > modified the FreeBSD driver to poll for Link in some local_timer > > > task. Link comes up but no traffic flows. The MAC statistics show > > > packets received but packets do not reach the guest. > > > DMAing of packet may be failing. I could not find out the reason. > > > The same happens with legacy interrupt allocated in the guest. I even > > > tried qemu-kvm prefer_msi=off. > > > > > > I think there are two problems1 > > > 1. Interrupt delivery either because of interrupt remapping failure or > > > all out interrupt allocation failure in qemu-kvm > > > 2. Packets not getting DMAed to the guest possibly because of some DMAR > > issue. > > > > > > Before doing VT-d I made sure connections are fine and the adapter > > > works fine in bare metal Linux. > > > I also tried VT-d of 82599 with Linux as guest (both 32 bits(PAE and > > > non-PAE) and amd64) and it just works magically. > > > > Unknowns: does the intel ixgbe driver work in FreeBSD with MSI-X > > > > I compiled Linux-3.2 with with VTD and used qemu-kcm-014.1. > > I tried the below cases with FreeBSD stock driver and Intel official > freebsd ixgbe-2.4.4 driver. > I tried using only msi interrupt (hw.pci.enable_msix="0" in l) it still did > not work. > I tried using legacy pin interrupt (hw.pci.enable_msi="0" in l) but it did > not work. > > The link was not coming up , the reason being no interrupts were received. > I modified ixgbe_local_timer() function in intel driver to call > ixgbe_check_link() > so that link status is updated periodically . Now the link came up but > still no traffic flow. > > My simple connectivity. > > Linux (ixgbe port) --------- (FreeBSD guest VTD ixgbe port) > > I added a static arp entry on Linux and did ping. The packets are seen by > the ixgbe > port of FreeBSD and I see ixgbe queue rx counts zero but > mac_stats.good_pkt_rcvd > shows the correct number of packets. This means the packets are reaching > the mac but are > not being sent to the h/w queue. > > There are three separate problems > 1. MSI-X Interrupts handling for FreeBSD is broken because of the way > FreeBSD allocates and update MSI-X tables. > 2. Interrupt generation and/or delivery to the guest is not happening. > 3. Packets received by mac are not DMAed and packets sent from FreBSD are > not DMA into the the ports memory. > > problem 2 and 3 symptoms point to a problem in FreeBSD Driver. > Problem 1 clearly seems to be a limitation on the qemu-kvm side which > expects > to receive MSI-X table in one shot. Yep, this is somewhat caused by the underlying MSIX interface that we work with. When we enable MSIX for the device we have to specify the number of vectors. We can either do that based on the table size specified in the MSIX capability or on the apparent number of vectors the driver tries to program. We currently do the latter both because it conserves unused MSI-X vectors and is usually a closer match to what the guest driver is trying to do. If we had better masking support we might be able to handle this better. Is this behavior unique to ixgbe freebsd driver or to anything with MSI-X on freebsd? Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
