Hi Alex,Jan,
I collected logs of pci updates processing of kvm(attached to this mail).
(I will try your suggestion soon)
The below source of Linux kernel shows the msix allocation done with
MSIX_ENABLE_FLAG
masked which works fine with kvm.
static int msix_capability_init(struct pci_dev *dev,
struct msix_entry *entries, int nvec)
{
int pos, ret;
u16 control;
void __iomem *base;
pos = pci_find_capability(dev, PCI_CAP_ID_MSIX);
pci_read_config_word(dev, pos + PCI_MSIX_FLAGS, &control);
/* Ensure MSI-X is disabled while it is set up */
control &= ~PCI_MSIX_FLAGS_ENABLE;
pci_write_config_word(dev, pos + PCI_MSIX_FLAGS, control);
/* Request & Map MSI-X table region */
base = msix_map_region(dev, pos, multi_msix_capable(control));
if (!base)
return -ENOMEM;
ret = msix_setup_entries(dev, pos, base, entries, nvec);
if (ret)
return ret;
ret = arch_setup_msi_irqs(dev, nvec, PCI_CAP_ID_MSIX);
if (ret)
/*
* Some devices require MSI-X to be enabled before we can touch the
* MSI-X registers. We need to mask all the vectors to prevent
* interrupts coming in before they're fully set up.
*/
control |= PCI_MSIX_FLAGS_MASKALL | PCI_MSIX_FLAGS_ENABLE;
pci_write_config_word(dev, pos + PCI_MSIX_FLAGS, control);
On Sat, Jan 14, 2012 at 3:45 AM, Jan Kiszka <jan.kiszka@xxxxxx> wrote:
> On 2012-01-13 22:56, Alex Williamson wrote:
>> On Fri, 2012-01-13 at 22:33 +0100, Jan Kiszka wrote:
>>> On 2012-01-13 22:05, Alex Williamson wrote:
>>>> On Fri, 2012-01-13 at 22:00 +0100, Jan Kiszka wrote:
>>>>> On 2012-01-04 04:21, Alex Williamson wrote:
>>>>>> On Mon, 2011-12-19 at 19:49 +0530, Shashidhar Patil wrote:
>>>>>>> Hi,
>>>>>>> I am running Ubuntu 10.10 (amd64) on a 2 socket nehalem based
>>>>>>> server with IOH 5520. 5520 supports VTD.
>>>>>>> I enabled DMAR with intel_iommu=on. The box has intel 82599 adapter
>>>>>>> which I assigned through VT-D to FreeBSD 8.2 running
>>>>>>> as guest os. The ixgbe driver detects the device and the driver
>>>>>>> successfully configures the device. But the link
>>>>>>> never comes up. It looks like link up/down interrupts are not
>>>>>>> delivered. Then I checked kvm interrupt assignment and as expected
>>>>>>> kvm could not make MSI-X entries for the VT-d guest. So no output from
>>>>>>> "grep kvm /proc/interrupt". By enabling some debugs in the
>>>>>>> qemu-kvm I figured out that the MSI-x updates are not received
>>>>>>> properly. It does look like Linux updates MSI-X table in a batch
>>>>>>> fashion
>>>>>>> which qemu-kvm gets in one shot and every thing works fine in case of
>>>>>>> linux. In case of FreeBSD PCIE updates come /MSI-X entry
>>>>>>> which qemu-kvm can't make use.
>>>>>>
>>>>>> That's right, Linux and Windows both seem to setup the MSI-X table then
>>>>>> enable it in one shot, so we only trigger the interrupt programming when
>>>>>> the enable bit is set. We don't trigger changes on writes to the MSI-X
>>>>>> table... not very accurate emulation of mask bits.
>>>>>
>>>>> According to the PCI spec, updates that happen while a vector is
>>>>> unmasked, need not be considered by the hardware (thus the hypervisor
>>>>> here). Is that the scenario here?
>>>>
>>>> I'm assuming the vector is masked in the MSI-X table. So Linux/Windows
>>>> do:
>>>>
>>>> a) program MSI-X table
>>>> b) enable MSI-X in capability register
>>>>
>>>> Whereas FreeBSD does:
>>>>
>>>> a) enable MSI-X in capability register (vectors masked in table)
>>>> b) program and unmask individual vectors
>>>
>>> That should work with the current code. It checks the number of vectors
>>> on each config write, iterates the whole table, and then updates the
>> ^^^^^^^^^^^^^^^^^^^^
>>> kernel configuration accordingly. It even requires the enable bit in the
>>> cap register to be set before doing this.
>>
>> That's the problem, we only do it on config writes overlapping the MSI-X
>> flags. We don't do anything for writes to the MSI-X table. It might be
>> as simple as calling assigned_dev_update_msix() from msix_mmio_writel()
>> when the mask bit is toggled. I'm not sure what might fall out of that
>> though.
>
> Ah indeed. Now I recall to have fixed this in my MSI-X refactoring
> series. I introduced config notifiers that are triggered by the MSI-X
> layer on every relevant modification, and the device assignment code
> hook the update function into this. I really need to dig into that
> series soon again and refresh it.
>
> In the meantime, we could try what you suggest (if the cap enable bit is
> set).
>
> Jan
>
Attachment:
kvm_bsd_msix.log
Description: Binary data
