RE: mptsas/iommu/pciehp : PCIe hotplug of LSISAS1064E fails withintel_iommu=on

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



> Yes - IOMMU support owns that. Anything allocated/mapped with
> pci_alloc_consistent() needs to have both read and write permissions
> for both Host and PCI target. This definitely not a device driver
> problem. Just need to verify the access was to that range.
> I've lost track of the original address that was the problem.

It was 0xfffc2000, which is in the range allocated (alloc_dma)

>>> I'd be looking for why the ioc number changed. It should be resuming
>> with the same instance number since it's the same physical device.

Yes , that's what I though as well.

I tried to trace the expected call sequence , it seems like pci resources are correctly unassigned.

The folling sequence is executed on removal:

pci_remove_bus_device
pci_stop_dev
MPT:detach
pci_destroy_devs
pci_free_resources

I figured out that the ioc 'id' is not decremented in mpt_detach() and that it's used for some kind of reverse lookup by mptsas.c and by mpt_verify_adapter(), mostly used for mptclt. But it appears to be almost only cosmetical...

...
On the IOMMU side, I see that it sets mapping (CONTEXT_TT_MULTI_LEVEL)) for the SAS controller.

#ipmitool  picmg policy set 2 1 0
deb64:~# pciehp 0000:00:04.0:pcie04: Card present on Slot(52)
pciehp 0000:00:04.0:pcie04: Latch close on Slot(52)
pciehp 0000:00:04.0:pcie04: Button pressed on Slot(52)
pciehp 0000:00:04.0:pcie04: PCI slot #52 - powering on due to button press.
Fusion MPT base driver 3.04.12
Copyright (c) 1999-2008 LSI Corporation
Fusion MPT SAS Host driver 3.04.12
mptsas 0000:06:00.0: enabling device (0000 -> 0002)
mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mptbase: ioc0: Initiating bringup
ioc0: LSISAS1064E B3: Capabilities={Initiator}
IOMMU: get valid domain for 0000:06:00.0
IOMMU: mapping 0000:06:00.0
IOMMU: lsi 06:00.0
IOMMU: continue mapping for 06:00.0
IOMMU: mapping for 06:00.0 completed
IOMMU: mapping no upstream  0000:06:00.0
scsi7 : ioc0: LSISAS1064E B3, FwRev=011b0000h, Ports=1, MaxQ=277, IRQ=16

...
I looked at the driver correctness for alloc/free of dma memory, and it seems the play by the book as long as the DMA goes. With some extra debugging I get:


ptbase: ioc0: ChainBuffer sz=128 bytes, ChainDepth=1143
mptbase: ioc0: ChainBuffer sz=146304[23b80] bytes num_chain=1143
IOMMU:alloc coherent
IOMMU:succeeded 000000003bc00000(ffff88003bc00000) 192512 ffffffff
mptbase: ioc0: Total alloc @ ffff88003bc00000[00000000fffc0000], sz=192000[2ee00] bytes
mptbase: ioc0: ReplyBuffers @ ffff88003bc00000[00000000fffc0000]
mptbase: ioc0: RequestBuffers @ ffff88003bc02800[00000000fffc2800]
mptbase: ioc0: ChainBuffers @ ffff88003bc0b280(00000000fffcb280)
IOMMU:alloc coherent
IOMMU:succeeded 000000003d1f8000(ffff88003d1f8000) 20480 ffffffff
mptbase: ioc0: SenseBuffers @ ffff88003d1f8000[00000000fffb8000]
mptbase: ioc0: ReplyBuffers @ ffff88003bc00000[00000000fffc0000]
mptbase: ioc0: SendIocInit
mptbase: ioc0: facts.MsgVersion=105
mptbase: ioc0: Sending Port(0)Enable (req @ ffff88003da6da98)
mptbase: ioc0: Wait IOC_OPERATIONAL state (cnt=0)

The DMA memory is freed on removal:

mptbase: ioc0: free  @ ffff88003bc00000, sz=192000 bytes
IOMMU:free coherent
IOMMU:free coherent
mptsas 0000:06:00.0: PCI INT A disabled

However, I see the 'free' called  at mptbase.c:4596 uses the allocated size (192000) rather than the page padded size(192512) really allocated by the IOMMU engine, but I doubt it could be the cause for this problem.
...

So I'm starting to think the IOMMU driver messes up the mapping tables in some way on deallocation/reallocation operations.

I must admist I don't fully understand the impact of this change: http://kerneltrap.org/mailarchive/git-commits-head/2009/7/3/6134073

But seeing these lines 
-			dma_set_pte_readable(pte);
-			dma_set_pte_writable(pte);

I can't believe it's not related to this message:

DMAR:[DMA Read] Request device [06:00.0] fault addr fffc3000
DMAR:[fault reason 06] PTE Read access is not set

... this still needs some investigation.

Thank you

François Isabelle | Software Designer | Kontron Canada | T 450 437 5682 |F 450 437 8053 | E francois.isabelle@xxxxxxxxxxxxxx

--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [SCSI Target Devel]     [Linux SCSI Target Infrastructure]     [Kernel Newbies]     [IDE]     [Security]     [Git]     [Netfilter]     [Bugtraq]     [Yosemite News]     [MIPS Linux]     [ARM Linux]     [Linux Security]     [Linux RAID]     [Linux ATA RAID]     [Linux IIO]     [Samba]     [Device Mapper]
  Powered by Linux