On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe <jgg@xxxxxxxx> wrote: > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > Can you please explain why it is so important to (allow) access them > > > > through the CPU ? > > > > > > It is not so much important, as it reflects significant design choices > > > that are already tightly baked into alot of our stacks. > > > > > > A SGL is CPU accessible by design - that is baked into this thing and > > > places all over the place assume it. Even in RDMA we have > > > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > > > to see) > > > > > > So, the thing at the top of the stack - in this case the gaudi driver > > > - simply can't assume what the rest of the stack is going to do and > > > omit the CPU side. It breaks everything. > > > > > > Logan's patch series is the most fully developed way out of this > > > predicament so far. > > > > I understand the argument and I agree that for the generic case, the > > top of the stack can't assume anything. > > Having said that, in this case the SGL is encapsulated inside a dma-buf object. > > > > Maybe its a stupid/over-simplified suggestion, but can't we add a > > property to the dma-buf object, > > that will be set by the exporter, which will "tell" the importer it > > can't use any CPU fallback ? Only "real" p2p ? > > The block stack has been trying to do something like this. > > The flag doesn't solve the DMA API/IOMMU problems though. hmm, I thought using dma_map_resource will solve the IOMMU issues, no ? We talked about it yesterday, and you said that it will "work" (although I noticed a tone of reluctance when you said that). If I use dma_map_resource to set the addresses inside the SGL before I export the dma-buf, and guarantee no one will use the SGL in the dma-buf for any other purpose than device p2p, what else is needed ? Oded > > Jason
