Thanks Grant for the info. On Thu, Feb 4, 2010 at 8:27 AM, Grant Grundler <grundler@xxxxxxxxxxxxxxxx> wrote: > On Fri, Jan 29, 2010 at 12:05:47PM +0530, yogeshwar sonawane wrote: >> Hi all, >> >> Following questions can be of naïve nature. Kindly, help me to clarify >> the points. >> Thanks in Advance. >> >> 1) Consider a situation where a PCIe device has requested a DMA Write >> operation followed by an MSI based interrupt. Upon receiving the >> interrupt, the software will read/check the DMAed data. Now, even if >> the device is guaranteeing the ordering between the two (issuing one >> after the other), whether the I/O controller OR chipset can >> reorder/buffer them? > > "can" == PCIe spec allows transactions to bypass DMA writes only in > one circumstance: Relaxed Ordering bit is set in the DMA write transaction. > Look at a PCIe trace to determine if that bit is set (or not). In this case, > due to the data dependency this bit should not be set. > >> This may result into getting interrupt before the >> data is DMAed resulting into software reading invalid data. Here data >> is residing into internal buffers of I/O controller and CPU is trying >> to read from memory. Whether such DMA reordering can happen/is allowed >> at all? Whether I/O controller can employ such intelligence(any kind >> of DMA reordering) for gaining some performance advantage? > > By "I/O Controller" do you mean the PCIe mother board chipset? > > I'm wondering if there is something else going on - e.g. non-cache > coherent DMA, bug in I/O controller causing DMA Write to get stuck > in a coalesce buffer or something like that. > > IIRC, tg3 driver lists examples of chipsets that do NOT > support PCI ordering correctly. > >> Mainly, >> focus is on high-end server class platforms having 8 or 16 cores with >> many peripherals. > > Which architecture? x86-64 CPUs and corresponding chipsets? IA64? Power? x86_64 arch. Mainly, i have not faced any problem, but i am wondering whether the problem can occur ? and how to handle that ? > >> 2) If above situation is at all possible, then to avoid such situation >> is the responsibility of the device or the driver for that device? >> Mainly, both the components (device and driver) expected to be generic >> in nature (platform, I/O controller independent). > > As noted, read Documentation/DMA-API.txt > >> 3) In continuation, if driver has to handle such situations, then how >> to handle such cases in Linux device driver? Any API is there for >> pending DMA flush/sync in I/O controller? I am aware of rmb(), wmb() & >> mb() which ensures ordering from CPU side i.e. slave read/writes. >> Anythings similar for DMA operations ? > > mmiowb() is also *required* on some platforms (e.g. SGI Altix) > > hth, > grant > >> Thanks and Regards, Yogeshwar -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
