On Monday, June 09, 2008 1:05 pm Matthew Wilcox wrote: > On Mon, Jun 09, 2008 at 12:00:48PM -0700, Meyers, Jordan wrote: > > During the process of writing a PCIe device driver, I observed an > > interesting phenomenon with respect to Transaction Layer Packet (TLP) > > generation. The device driver needs to transfer upwards of 1/3 KB in > > the payload of a single TLP.? I used the pci_iomap function combined > > with memcpy_toio for data transfer to the address space pointed to > > by a Base Address Register (BAR).? When I used the memcpy_toio, the > > buffer content I tried to write was split into dword (4B) sized pieces > > and each dword was given its own TLP.? According to PCI-SIG and some of > > the research I've done, the PCIe Bus Controller is responsible for this > > behavior. The PCIe Bus Controller turns the driver accesses into TLPs. > > This presents an interesting problem. Since the generation of these TLPs > > (from the BAR region) are out of the hands of the Operating System, > > what procedure can a driver go through to ensure that all the data > > makes it into a single TLP payload. One thought I had was the use of > > DMA buffers, and having the PCIe card pull the data from the DMA buffer. > > I observed the DMA pushes from the card to the buffer, were not subject > > to the same limiting factors. > > While the PCI spec would say that it was the responsibility of the bus > controller, that's because it can't talk meaningfully about the rest of > the system. In your case, the problem is that you have the mmio area > mapped uncachable (since we don't know what semantics the device has for > this region, we default to the conservative assumption that it's full of > registers). So the processor ends up emitting 4-byte writes to the bus > controller (since it hasn't been told that it can combine writes, it > doesn't). The bus controller is likewise helpless to combine writes and > so streams a sequence of 4-byte writes. > > You can increase the likelihood of generating full writes (but not > guarantee it) by asking for a write-combining mapping of this area > with ioremap_cache(). If this is a card you're developing, you should > probably set the 'prefetchable' bit in the BAR in question. Yeah, that should help, but probably still won't be as big as you want (depends on the size of the write combine buffers in your CPU). I think the only way to get the sort of behavior you want would be to use a special chipset with some sort of host data upload buffer & trigger mechanism, e.g. write target address to chipset write data to chipset write trigger value to chipset which would allow the chipset to send a large transaction; alternately you could have your device use DMA to read the data from memory directly rather than trying to write it with the CPU. Jesse -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
