On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > Can you please explain why it is so important to (allow) access them > > through the CPU ? > > It is not so much important, as it reflects significant design choices > that are already tightly baked into alot of our stacks. > > A SGL is CPU accessible by design - that is baked into this thing and > places all over the place assume it. Even in RDMA we have > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > to see) > > So, the thing at the top of the stack - in this case the gaudi driver > - simply can't assume what the rest of the stack is going to do and > omit the CPU side. It breaks everything. > > Logan's patch series is the most fully developed way out of this > predicament so far. I understand the argument and I agree that for the generic case, the top of the stack can't assume anything. Having said that, in this case the SGL is encapsulated inside a dma-buf object. Maybe its a stupid/over-simplified suggestion, but can't we add a property to the dma-buf object, that will be set by the exporter, which will "tell" the importer it can't use any CPU fallback ? Only "real" p2p ? Won't that solve the problem by eliminating the unsupported access methods ? Oded > > > The whole purpose is that the other device accesses my device, > > bypassing the CPU. > > Sure, but you don't know that will happen, or if it is even possible > in any given system configuration. The purpose is to allow for that > optimization when possible, not exclude CPU based approaches. > > Jason
