On Fri, Jan 29, 2010 at 12:05:47PM +0530, yogeshwar sonawane wrote: > Hi all, > > Following questions can be of naïve nature. Kindly, help me to clarify > the points. > Thanks in Advance. > > 1) Consider a situation where a PCIe device has requested a DMA Write > operation followed by an MSI based interrupt. Upon receiving the > interrupt, the software will read/check the DMAed data. Now, even if > the device is guaranteeing the ordering between the two (issuing one > after the other), whether the I/O controller OR chipset can > reorder/buffer them? "can" == PCIe spec allows transactions to bypass DMA writes only in one circumstance: Relaxed Ordering bit is set in the DMA write transaction. Look at a PCIe trace to determine if that bit is set (or not). In this case, due to the data dependency this bit should not be set. > This may result into getting interrupt before the > data is DMAed resulting into software reading invalid data. Here data > is residing into internal buffers of I/O controller and CPU is trying > to read from memory. Whether such DMA reordering can happen/is allowed > at all? Whether I/O controller can employ such intelligence(any kind > of DMA reordering) for gaining some performance advantage? By "I/O Controller" do you mean the PCIe mother board chipset? I'm wondering if there is something else going on - e.g. non-cache coherent DMA, bug in I/O controller causing DMA Write to get stuck in a coalesce buffer or something like that. IIRC, tg3 driver lists examples of chipsets that do NOT support PCI ordering correctly. > Mainly, > focus is on high-end server class platforms having 8 or 16 cores with > many peripherals. Which architecture? x86-64 CPUs and corresponding chipsets? IA64? Power? > 2) If above situation is at all possible, then to avoid such situation > is the responsibility of the device or the driver for that device? > Mainly, both the components (device and driver) expected to be generic > in nature (platform, I/O controller independent). As noted, read Documentation/DMA-API.txt > 3) In continuation, if driver has to handle such situations, then how > to handle such cases in Linux device driver? Any API is there for > pending DMA flush/sync in I/O controller? I am aware of rmb(), wmb() & > mb() which ensures ordering from CPU side i.e. slave read/writes. > Anythings similar for DMA operations ? mmiowb() is also *required* on some platforms (e.g. SGI Altix) hth, grant > > Kindly, update me if i am missing something. > > Any info,link or reference will be of great help. > > Thanks and Regards, > Yogeshwar > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@xxxxxxxxxxxxxxx > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
