On Mon, 8 Jan 2018 08:26:46 +0100 Binarus <lists@xxxxxxxxxx> wrote: > On 07.01.2018 07:22, Alex Williamson wrote: > > > For the system you're using there is no integrated graphics on the > > processor, so "igfx_off" does nothing. However, using that as the only > > parameter to intel_iommu= is basically the same as not providing an > > intel_iommu= option at all, ie. the IOMMU is not enabled, thus... > > At first, thank you very much for spending time on that problem! > > I see. This is the opposite logic than what I would have expected :-), > and it explains the behavior (I would have expected that the option > would be turned on if encountered in any case, and that inappropriate > option parameters would be ignored instead of turning the option off). > > Additionally, I have misunderstood the following passage > > "If you encounter issues with graphics devices, you can try adding > option intel_iommu=igfx_off to turn off the integrated graphics engine. > If this fixes anything, please ensure you file a bug reporting the problem." > > from the following document: > > https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt > > The wording "If this fixes anything" made me somehow think that igfx_off > in rare cases might do something mysterious even to devices which are > not integrated graphics devices. > > Thank you very much for clearing that up and for explaining how > intel_iommu works. > > >> This message is to report that I can't pass through the SATA controller in question to a VM when the kernel has been booted with intel_iommu=igfx_off. Things are going wrong already during the boot process. dmesg | grep vfio says: > >> > >> [ 1.074239] vfio-pci: probe of 0000:02:00.0 failed with error -22 > >> [ 1.074309] vfio_pci: add [1b4b:9128[ffff:ffff]] class 0x000000/00000000 > > > > vfio-pci won't probe the device because it doesn't have an IOMMU. > > This is Murphy's law. I have thought a long time about buying another > controller with another chipset and obviously made the wrong choice. I > should have read the controller's datasheet before buying, but > understanding it would surely have cost me a huge amount of time. The > controller was about 30 EUR (about 35 US$), so spending two days on > understanding that datasheet wouldn't have made a lot of sense ... I'm not even sure Marvell has figured out that they have this problem, let alone documented it in a datasheet ;) > > Marvell SATA controllers are notoriously bad with IOMMUs, they perform > > DMA using a requester ID other than the actual PCI address of the > > device. For instance, the SATA controller is often at PCI function > > address 0, but function address 1 is used for the requester ID. The > > IOMMU uses the requester ID to index the DMA translations for the > > device, so getting this fundamental aspect of the transaction wrong in > > hardware is often fatal for doing any DMA at all when the IOMMU is > > enabled. This should cause all sorts of "DMAR" faults during boot when > > VT-d is enabled. > > Thanks for the in-depth explanation of the problem. I have never read > the Marvell datasheets nor the IOMMU specs until now. I wish I had some > more time; that whole thing is very interesting. > > > We do have quirks to handle various Marvell chips, but not this one. > > If you're interested in building a kernel, I can provide a patch to > > test, though I suspect the reason we don't already have a patch for > > this device might be conflicting results. Your other option is to pick > > a SATA controller that isn't so broken. Thanks, > > Initially, I didn't want to build a new kernel on that machine (I don't > have a second one at hands with identical hardware). > > But I am quite impressed by the KVM project in general, and that you > already can offer a patch for testing in particular, so I'll do it. I > would be glad if I could be of any help to the KVM project. Could you > please provide me instructions? Can your patch be applied to 4.8 or 4.9, > or only to the newest kernel? We already have quirks to support various other versions of the Marvell chip, but the 9128 is missing, so it's just a couple lines to add it. This is against v4.9.75: diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c index 98eba9127a0b..19ca3c9fac3a 100644 --- a/drivers/pci/quirks.c +++ b/drivers/pci/quirks.c @@ -3868,6 +3868,8 @@ static void quirk_dma_func1_alias(struct pci_dev *dev) /* https://bugzilla.kernel.org/show_bug.cgi?id=42679#c49 */ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9230, quirk_dma_func1_alias); +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9128, + quirk_dma_func1_alias); DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_TTI, 0x0642, quirk_dma_func1_alias); /* https://bugs.gentoo.org/show_bug.cgi?id=497630 */ > One additional question (to get my problem solved in any case): Could > you recommend a SATA controller (chip) which works flawlessly in that > respect? I just need at least one SATA channel (but better two) passed > through into that VM - no raid, mirroring, striping or other fancy > things needed. Personally, I don't often assign storage controllers and they're mostly all terrible. The Marvell controllers nearly all have this DMA aliasing issue afaik, but you can see in the code nearby the patch above that we already have quirks for many of them. For instance you could buy a similarly priced Marvell 9230 card rather than a 9128 and it might have worked since we've already got a quirk for it. Sorry I can't be more precise, even as the device assignment maintainer I generally use virtio for VM disks and find it to be sufficiently fast and feature-ful. Thanks, Alex
