>>PrimeIocFifos calls pci_alloc_consistent() and has debug code to dump
>>the DMA resource allocated (both virtual and DMA addresses). Off hand
>>I don't know how to enable that but it would be the next step.
With mpt_debug_level=0x8060 , I was able to get the allocation code to be more verbose, see below.
>>> DRHD: handling fault status reg 2
>>> DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000
>>fffc2000 seems to be an unusual address to DMA from/to.
>>Is fffc2000 reserved address space for the IOMMU?
>> (ACPI DMAR info should tell us this)
It seems to be the actual address (see below); however I think there might be some kind of page overlap between the 'reply buffer' and 'request buffer' ...
mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]
> > > > DMAR:[fault reason 06] PTE Read access is not set
> > > It's also odd that "Read Access is not set" for something (ioc_init)
> > > that I think should be bi-directional.
Wouldn't that depend on the IOMMU driver? Or is the device driver supposed to set protections?
When we compare the 2 'init' we see differences in the 'mem_phys' entry, which I think is weird, especially if it should match the PCI memory region of the device
>> Region 1: Memory at dbffc000 (64-bit, non-prefetchable) [size=16K]
>> Region 3: Memory at dbfe0000 (64-bit, non-prefetchable) [size=64K]
mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
...
mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c
..and we see differences in the chain buffers..
mptbase: ioc0: ReqToChain alloc @ ffff88003d4d8800, sz=1108 bytes
mptbase: ioc0: RequestNB alloc @ ffff88003e3f3000, sz=1108 bytes
mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes
mptbase: ioc1: ReqToChain alloc @ ffff88003d2aa000, sz=1108 bytes
mptbase: ioc1: RequestNB alloc @ ffff88003d268800, sz=1108 bytes
mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes
The other thing that is now obvious is that the device name has changed (ioc1 instead of ioc0); this makes me think that the whole problem is probably more closely related to the resources unallocation, maybe a missing 'unlink' operation of some kind... I'm looking into this, but I could still use some help if anyone has an idea.
Following is the complete MPTSAS init log.
==== First time init ===
mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mptbase: ioc0: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
mptbase: ioc0: Initiating bringup
mptbase: ioc0: IOC is in READY state
mptbase: ioc0: Sending get IocFacts request req_sz=12 reply_sz=80
mptbase: ioc0: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
mptbase: ioc0: reply_sz= 80, reply_depth= 128
mptbase: ioc0: req_sz =128, req_depth = 277
mptbase: ioc0: Sending get PortFacts(0) request
ioc0: LSISAS1064E B3: Capabilities={Initiator}
mptsas 0000:06:00.0: setting latency timer to 64
mptbase: ioc0: installed at interrupt 16
mptbase: ioc0: PrimeIocFifos
mptbase: ioc0: ReqToChain alloc @ ffff88003d4d8800, sz=1108 bytes
mptbase: ioc0: RequestNB alloc @ ffff88003e3f3000, sz=1108 bytes
mptbase: ioc0: num_sge=25 numSGE=520
mptbase: ioc0: Now numSGE=128 num_sge=130 num_chain=9
mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes
mptbase: ioc0: ReplyBuffer sz=80 bytes, ReplyDepth=128
mptbase: ioc0: ReplyBuffer sz=10240[2800] bytes
mptbase: ioc0: RequestBuffer sz=128 bytes, RequestDepth=277
mptbase: ioc0: RequestBuffer sz=35456[8a80] bytes
mptbase: ioc0: ChainBuffer sz=128 bytes, ChainDepth=1143
mptbase: ioc0: ChainBuffer sz=146304[23b80] bytes num_chain=1143
mptbase: ioc0: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]
mptbase: ioc0: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
mptbase: ioc0: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc0: SendIocInit
mptbase: ioc0: facts.MsgVersion=105
mptbase: ioc0: Sending Port(0)Enable (req @ ffff88003cc63a98)
mptbase: ioc0: Wait IOC_OPERATIONAL state (cnt=0)
mptbase: ioc0: SendEventNotification
scsi15 : ioc0: LSISAS1064E B3, FwRev=011b0000h, Ports=1, MaxQ=277, IRQ=16
==== Reinit after PCIe Hotplug ===
tsas 0000:06:00.0: enabling device (0000 -> 0002)
mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
mptbase: ioc1: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c
mptbase: ioc1: Initiating bringup
mptbase: ioc1: IOC is in READY state
mptbase: ioc1: Sending get IocFacts request req_sz=12 reply_sz=80
mptbase: ioc1: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
mptbase: ioc1: reply_sz= 80, reply_depth= 128
mptbase: ioc1: req_sz =128, req_depth = 277
mptbase: ioc1: Sending get PortFacts(0) request
ioc1: LSISAS1064E B3: Capabilities={Initiator}
mptsas 0000:06:00.0: setting latency timer to 64
mptbase: ioc1: installed at interrupt 16
mptbase: ioc1: PrimeIocFifos
mptbase: ioc1: ReqToChain alloc @ ffff88003d2aa000, sz=1108 bytes
mptbase: ioc1: RequestNB alloc @ ffff88003d268800, sz=1108 bytes
mptbase: ioc1: num_sge=25 numSGE=520
mptbase: ioc1: Now numSGE=128 num_sge=130 num_chain=9
mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes
mptbase: ioc1: ReplyBuffer sz=80 bytes, ReplyDepth=128
mptbase: ioc1: ReplyBuffer sz=10240[2800] bytes ==========> (2800h bytes)
mptbase: ioc1: RequestBuffer sz=128 bytes, RequestDepth=277
mptbase: ioc1: RequestBuffer sz=35456[8a80] bytes
mptbase: ioc1: ChainBuffer sz=128 bytes, ChainDepth=1143
mptbase: ioc1: ChainBuffer sz=146304[23b80] bytes num_chain=1143
mptbase: ioc1: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
mptbase: ioc1: RequestBuffers @ ffff88003b182800[00000000fffc2800]
mptbase: ioc1: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
mptbase: ioc1: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000] ==========> (2800h bytes)
mptbase: ioc1: SendIocInit
mptbase: ioc1: facts.MsgVersion=105
mptbase: ioc1: Sending Port(0)Enable (req @ ffff88003e28d880)
deb64:~#
b64:~# mptbase: ioc1: WARNING - Issuing Reset from mpt_config!!
mptbase: ioc1: Initiating recovery
mptbase: ioc1: WARNING - IOC is in FAULT state!!!
mptbase: ioc1: WARNING - FAULT code = 2000h
mptbase: ioc1: Recovered from IOC FAULT
mptbase: ioc1: PrimeIocFifos
mptbase: ioc1: SendIocInit
mptbase: ioc1: SendEventNotification
mptbase: ioc1: Attempting Retry Config request type 0x1, page 0x2, action 0
DRHD: handling fault status reg 102
DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000
DMAR:[fault reason 06] PTE Read access is not set
Thank you
François Isabelle | Software Designer | Kontron Canada | T 450 437 5682 |F 450 437 8053 | E francois.isabelle@xxxxxxxxxxxxxx
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
