Matthew Wilcox <matthew@xxxxxx> writes: > On Mon, Nov 29, 2010 at 11:43:16AM -0500, Nikolaus Rath wrote: >> I would like to transfer data between a PXI card (a digitizer, to be >> precise) and a PCI express card (an Nvidia Tesla GPU) as fast as >> possible. >> >> Is it possible to have two cards write into each others memory directly, >> without ever involving the host CPU? If I wanted to implement this, >> where would I have to start? Do I need to go to kernel space? Or are >> there API functions that I can use from userspace? > > You'll have to go to kernel space, but yes, it's possible. Not all > chipsets support this functionality (peer to peer transfers) efficiently, > and it may be more efficient for you to write to host memory and then > read from host memory. My main concern is latency. Input + processing (by the GPU) + output of a data set should take in the order of 20 us, so my feeling is that I should rather avoid the host memory. Can you tell me how to find out if a given chipset has the required functionality? Is there a technical term for that I could look for in the specs, or is it reported somewhere in /sys or /proc when I have installed the card? >> I already found out that I can connect the PXI bus to either the PCI or >> PCI Express bus of the host computer using a bus extender. Is there a >> difference between these two variants as far as communicating with other >> PCIe cards is concerned? > > It depends very much on the chipset involved. It probably doesn't > make a big difference since you'll be going through a PCI<->PCIe bridge > either way. Ah, so if the chipsets of the two cards support device to device transfers, then it will be possible no matter how I join the buses? Wouldn't the pxi -> bus extender -> pci -> bridge -> PCIe path involve more latency than the pxi -> bus extender -> PCIe path? Can you give me any advice how to find out how good or bad a given chipset will do? What are the criteria that I should pay attention to? >> Finally, can someone give me an order-of-magnitude estimate of how the >> latencies that will be involved in such a transfer? How long will it >> take to transfer 90 16bit samples from one card to the other? > > The latency will all be in the setup. You only want to transfer 180 > bytes, and even the 33MHz, 32bit PCI bus can transfer that much data > in under two microseconds. Yes, that's what I expected. But what is a reasonable time for the setup? Thanks for all the information, this already helps a lot! Best, -Nikolaus -- ÂTime flies like an arrow, fruit flies like a Banana. PGP fingerprint: 5B93 61F8 4EA2 E279 ABF6 02CF A9AD B7F8 AE4E 425C -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
