During the process of writing a PCIe device driver, I observed an interesting phenomenon with respect to Transaction Layer Packet (TLP) generation. The device driver needs to transfer upwards of 1/3 KB in the payload of a single TLP. I used the pci_iomap function combined with memcpy_toio for data transfer to the address space pointed to by a Base Address Register (BAR). When I used the memcpy_toio, the buffer content I tried to write was split into dword (4B) sized pieces and each dword was given its own TLP. According to PCI-SIG and some of the research I've done, the PCIe Bus Controller is responsible for this behavior. The PCIe Bus Controller turns the driver accesses into TLPs. This presents an interesting problem. Since the generation of these TLPs (from the BAR region) are out of the hands of the Operating System, what procedure can a driver go through to ensure that all the data makes it into a single TLP payload. One thought I had was the use of DMA buffers, and having the PCIe card pull the data from the DMA buffer. I observed the DMA pushes from the card to the buffer, were not subject to the same limiting factors.
All the observations I made were done by using a Lecroy PCIe analyzer. The machine I ran this on was a Dell Precision 490 (which had 2 AMD quad core, x86_64 architecture). I've tested the behavior on SUSE Linux running kernel 2.6.16 and 2.6.22. And the chipset was Intel 5000x ( the relevant PCIe component of the chipset is the 6321 ESB I/O Hub Controller).
Any thoughts or suggestions of how to ensure a buffer's data is transferred from the driver to the PCIe card in a single TLP (where the intended payload size is less than the max payload value in Device Control register of the PCI register space)? Thank you to everyone who has taken the time to read this post.
Jordan Meyers
--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in
the body of a message to majordomo@xxxxxxxxxxxxxxx
More majordomo info at http://vger.kernel.org/majordomo-info.html
