On Wed, Sep 16, 2009 at 12:04:27PM -0400, Isabelle, Francois wrote:
> >>PrimeIocFifos calls pci_alloc_consistent() and has debug code to dump
> >>the DMA resource allocated (both virtual and DMA addresses). Off hand
> >>I don't know how to enable that but it would be the next step.
>
> With mpt_debug_level=0x8060 , I was able to get the allocation code to be more verbose, see below.
Cool - noted.
> >>> DRHD: handling fault status reg 2
> >>> DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000
>
> >>fffc2000 seems to be an unusual address to DMA from/to.
> >>Is fffc2000 reserved address space for the IOMMU?
> >> (ACPI DMAR info should tell us this)
>
> It seems to be the actual address (see below); however I think there might be some kind of page overlap between the 'reply buffer' and 'request buffer' ...
>
> mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]
Excellent. This corresponds with:
dinitprintk(ioc, printk(MYIOC_s_DEBUG_FMT "ReplyBuffers @ %p[%p]\n",
ioc->name, ioc->reply_frames, (void *)(ulong)alloc_dma));
alloc_dma is the DMA Mapped address.
I don't believe the two ranges overlap based on this output:
> mptbase: ioc0: ReplyBuffer sz=80 bytes, ReplyDepth=128
> mptbase: ioc0: ReplyBuffer sz=10240[2800] bytes
that you included at the bottom of the email.
>
> > > > > DMAR:[fault reason 06] PTE Read access is not set
> > > > It's also odd that "Read Access is not set" for something (ioc_init)
> > > > that I think should be bi-directional.
>
> Wouldn't that depend on the IOMMU driver? Or is the device driver supposed to set protections?
Yes - IOMMU support owns that. Anything allocated/mapped with
pci_alloc_consistent() needs to have both read and write permissions
for both Host and PCI target. This definitely not a device driver
problem. Just need to verify the access was to that range.
I've lost track of the original address that was the problem.
> When we compare the 2 'init' we see differences in the 'mem_phys' entry, which I think is weird, especially if it should match the PCI memory region of the device
> >> Region 1: Memory at dbffc000 (64-bit, non-prefetchable) [size=16K]
> >> Region 3: Memory at dbfe0000 (64-bit, non-prefetchable) [size=64K]
>
> mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
> mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
> ...
> mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
> mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c
The Region 1 and Region 3 are for the same controller (ioc0).
Not for two different controllers. The second controller should
have it's own pair of Region 1 and Region 3 configuration space registers.
> ..and we see differences in the chain buffers..
>
> mptbase: ioc0: ReqToChain alloc @ ffff88003d4d8800, sz=1108 bytes
> mptbase: ioc0: RequestNB alloc @ ffff88003e3f3000, sz=1108 bytes
> mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes
>
> mptbase: ioc1: ReqToChain alloc @ ffff88003d2aa000, sz=1108 bytes
> mptbase: ioc1: RequestNB alloc @ ffff88003d268800, sz=1108 bytes
> mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes
I'm not sure what you are comparing with the previous 6 lines of output.
The two controller (ioc0 and ioc1) are required to have different
chain buffers.
> The other thing that is now obvious is that the device name has
> changed (ioc1 instead of ioc0);
Yes - good catch.
> this makes me think that the whole
> problem is probably more closely related to the resources unallocation,
> maybe a missing 'unlink' operation of some kind... I'm looking into
> this, but I could still use some help if anyone has an idea.
I'd be looking for why the ioc number changed. It should be resuming
with the same instance number since it's the same physical device.
hth,
grant
>
> Following is the complete MPTSAS init log.
>
> ==== First time init ===
>
> mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> mptbase: ioc0: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
> mptbase: ioc0: mem = ffffc90012e40000, mem_phys = dbffc000
> mptbase: ioc0: facts @ ffff88003dc2541c, pfacts[0] @ ffff88003dc2546c
> mptbase: ioc0: Initiating bringup
> mptbase: ioc0: IOC is in READY state
> mptbase: ioc0: Sending get IocFacts request req_sz=12 reply_sz=80
> mptbase: ioc0: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
> mptbase: ioc0: reply_sz= 80, reply_depth= 128
> mptbase: ioc0: req_sz =128, req_depth = 277
> mptbase: ioc0: Sending get PortFacts(0) request
> ioc0: LSISAS1064E B3: Capabilities={Initiator}
> mptsas 0000:06:00.0: setting latency timer to 64
> mptbase: ioc0: installed at interrupt 16
> mptbase: ioc0: PrimeIocFifos
> mptbase: ioc0: ReqToChain alloc @ ffff88003d4d8800, sz=1108 bytes
> mptbase: ioc0: RequestNB alloc @ ffff88003e3f3000, sz=1108 bytes
> mptbase: ioc0: num_sge=25 numSGE=520
> mptbase: ioc0: Now numSGE=128 num_sge=130 num_chain=9
> mptbase: ioc0: ChainToChain alloc @ ffff88003db14000, sz=4572 bytes
> mptbase: ioc0: ReplyBuffer sz=80 bytes, ReplyDepth=128
> mptbase: ioc0: ReplyBuffer sz=10240[2800] bytes
> mptbase: ioc0: RequestBuffer sz=128 bytes, RequestDepth=277
> mptbase: ioc0: RequestBuffer sz=35456[8a80] bytes
> mptbase: ioc0: ChainBuffer sz=128 bytes, ChainDepth=1143
> mptbase: ioc0: ChainBuffer sz=146304[23b80] bytes num_chain=1143
> mptbase: ioc0: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
> mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc0: RequestBuffers @ ffff88003b182800[00000000fffc2800]
> mptbase: ioc0: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
> mptbase: ioc0: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
> mptbase: ioc0: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc0: SendIocInit
> mptbase: ioc0: facts.MsgVersion=105
> mptbase: ioc0: Sending Port(0)Enable (req @ ffff88003cc63a98)
> mptbase: ioc0: Wait IOC_OPERATIONAL state (cnt=0)
> mptbase: ioc0: SendEventNotification
> scsi15 : ioc0: LSISAS1064E B3, FwRev=011b0000h, Ports=1, MaxQ=277, IRQ=16
>
> ==== Reinit after PCIe Hotplug ===
>
> tsas 0000:06:00.0: enabling device (0000 -> 0002)
> mptsas 0000:06:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
> mptbase: ioc1: : 32 BIT PCI BUS DMA ADDRESSING SUPPORTED
> mptbase: ioc1: mem = ffffc90012e48000, mem_phys = d8010000
> mptbase: ioc1: facts @ ffff88003d21041c, pfacts[0] @ ffff88003d21046c
> mptbase: ioc1: Initiating bringup
> mptbase: ioc1: IOC is in READY state
> mptbase: ioc1: Sending get IocFacts request req_sz=12 reply_sz=80
> mptbase: ioc1: NB_for_64_byte_frame=2 NBShiftFactor=5 BlockSize=8
> mptbase: ioc1: reply_sz= 80, reply_depth= 128
> mptbase: ioc1: req_sz =128, req_depth = 277
> mptbase: ioc1: Sending get PortFacts(0) request
> ioc1: LSISAS1064E B3: Capabilities={Initiator}
> mptsas 0000:06:00.0: setting latency timer to 64
> mptbase: ioc1: installed at interrupt 16
> mptbase: ioc1: PrimeIocFifos
> mptbase: ioc1: ReqToChain alloc @ ffff88003d2aa000, sz=1108 bytes
> mptbase: ioc1: RequestNB alloc @ ffff88003d268800, sz=1108 bytes
> mptbase: ioc1: num_sge=25 numSGE=520
> mptbase: ioc1: Now numSGE=128 num_sge=130 num_chain=9
> mptbase: ioc1: ChainToChain alloc @ ffff88003dfda000, sz=4572 bytes
> mptbase: ioc1: ReplyBuffer sz=80 bytes, ReplyDepth=128
> mptbase: ioc1: ReplyBuffer sz=10240[2800] bytes ==========> (2800h bytes)
> mptbase: ioc1: RequestBuffer sz=128 bytes, RequestDepth=277
> mptbase: ioc1: RequestBuffer sz=35456[8a80] bytes
> mptbase: ioc1: ChainBuffer sz=128 bytes, ChainDepth=1143
> mptbase: ioc1: ChainBuffer sz=146304[23b80] bytes num_chain=1143
> mptbase: ioc1: Total alloc @ ffff88003b180000[00000000fffc0000], sz=192000[2ee00] bytes
> mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000]
> mptbase: ioc1: RequestBuffers @ ffff88003b182800[00000000fffc2800]
> mptbase: ioc1: ChainBuffers @ ffff88003b18b280(00000000fffcb280)
> mptbase: ioc1: SenseBuffers @ ffff88003d2f8000[00000000fffb8000]
> mptbase: ioc1: ReplyBuffers @ ffff88003b180000[00000000fffc0000] ==========> (2800h bytes)
> mptbase: ioc1: SendIocInit
> mptbase: ioc1: facts.MsgVersion=105
> mptbase: ioc1: Sending Port(0)Enable (req @ ffff88003e28d880)
> deb64:~#
>
>
> b64:~# mptbase: ioc1: WARNING - Issuing Reset from mpt_config!!
> mptbase: ioc1: Initiating recovery
> mptbase: ioc1: WARNING - IOC is in FAULT state!!!
> mptbase: ioc1: WARNING - FAULT code = 2000h
> mptbase: ioc1: Recovered from IOC FAULT
> mptbase: ioc1: PrimeIocFifos
> mptbase: ioc1: SendIocInit
> mptbase: ioc1: SendEventNotification
> mptbase: ioc1: Attempting Retry Config request type 0x1, page 0x2, action 0
> DRHD: handling fault status reg 102
> DMAR:[DMA Read] Request device [06:00.0] fault addr fffc2000
> DMAR:[fault reason 06] PTE Read access is not set
>
>
> Thank you
>
> François Isabelle | Software Designer | Kontron Canada | T 450 437 5682 |F 450 437 8053 | E francois.isabelle@xxxxxxxxxxxxxx
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
