On Mon, Jun 09, 2008 at 12:00:48PM -0700, Meyers, Jordan wrote: > > During the process of writing a PCIe device driver, I observed an > interesting phenomenon with respect to Transaction Layer Packet (TLP) > generation. The device driver needs to transfer upwards of 1/3 KB in > the payload of a single TLP.? I used the pci_iomap function combined > with memcpy_toio for data transfer to the address space pointed to > by a Base Address Register (BAR).? When I used the memcpy_toio, the > buffer content I tried to write was split into dword (4B) sized pieces > and each dword was given its own TLP.? According to PCI-SIG and some of > the research I've done, the PCIe Bus Controller is responsible for this > behavior. The PCIe Bus Controller turns the driver accesses into TLPs. > This presents an interesting problem. Since the generation of these TLPs > (from the BAR region) are out of the hands of the Operating System, > what procedure can a driver go through to ensure that all the data > makes it into a single TLP payload. One thought I had was the use of > DMA buffers, and having the PCIe card pull the data from the DMA buffer. > I observed the DMA pushes from the card to the buffer, were not subject > to the same limiting factors. While the PCI spec would say that it was the responsibility of the bus controller, that's because it can't talk meaningfully about the rest of the system. In your case, the problem is that you have the mmio area mapped uncachable (since we don't know what semantics the device has for this region, we default to the conservative assumption that it's full of registers). So the processor ends up emitting 4-byte writes to the bus controller (since it hasn't been told that it can combine writes, it doesn't). The bus controller is likewise helpless to combine writes and so streams a sequence of 4-byte writes. You can increase the likelihood of generating full writes (but not guarantee it) by asking for a write-combining mapping of this area with ioremap_cache(). If this is a card you're developing, you should probably set the 'prefetchable' bit in the BAR in question. -- Intel are signing my paycheques ... these opinions are still mine "Bill, look, we understand that you're interested in selling us this operating system, but compare it to ours. We can't possibly take such a retrograde step." -- To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
