Haggi > I'd be happy to see your RFC when you are ready. I see in the thread of [3] > that you are using write-combining. Do you think your patchset will also be > suitable for uncachable memory? Great, we hope to have the RFC soon. It will be able to accept different flags for devm_memremap() call with regards to caching. Though one question I have is when does the caching flag affect Peer-2-Peer memory accesses? I can see caching causing issues when performing accesses from the CPU but P2P accesses should bypass any caches in the system? > I don't think that's enough for our purposes. We have devices with rather > small BARs (32MB) and multiple PFs that all need to expose their BAR to peer > to peer access. One can expect these PFs will be assigned adjacent addresses > and they will break the "one dev_pagemap per section" rule. On the cards and systems I have checked even small BARs tend to be separated by more than one section's worth of memory. As I understand it the allocation of BAR addresses is very ARCH and BIOS specific. Let's discuss this once the RFC comes out and see what options exist to address your concerns. > > > 4. The out of tree patch we did allows one to register the device memory as > IO memory. However, we were only concerned with DRAM exposed on the > BAR and so were not affected by the "i/o side effects" issues. Someone > would need to think about how this applies to IOMEM that does have side- > effects when accessed. > With this RFC, we map parts of the HCA BAR that were mmapped to a > process (both uncacheable and write-combining) and map them to a peer > device (another HCA). As long as the kernel doesn't do anything else with > these pages, and leaves them to be controlled by the user-space application > and/or the peer device, I don't see a problem with mapping IO memory with > side effects. However, I'm not an expert here, and I'd be happy to hear what > others think about this. See above. I think the upcoming RFC should provide support for both caching and uncashed mappings. I concur that even if the mappings are flagged as cachable there should be no issues as long as all accesses are from the peer-direct device. -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
