On Tue, Jun 30, 2020 at 08:08:46PM +0000, Xiong, Jianxin wrote: > > From: Jason Gunthorpe <jgg@xxxxxxxx> > > Sent: Tuesday, June 30, 2020 12:17 PM > > To: Xiong, Jianxin <jianxin.xiong@xxxxxxxxx> > > Cc: linux-rdma@xxxxxxxxxxxxxxx; Doug Ledford <dledford@xxxxxxxxxx>; Sumit Semwal <sumit.semwal@xxxxxxxxxx>; Leon Romanovsky > > <leon@xxxxxxxxxx>; Vetter, Daniel <daniel.vetter@xxxxxxxxx>; Christian Koenig <christian.koenig@xxxxxxx>; dri- > > devel@xxxxxxxxxxxxxxxxxxxxx > > Subject: Re: [RFC PATCH v2 0/3] RDMA: add dma-buf support > > > > > > > > > > On Tue, Jun 30, 2020 at 05:21:33PM +0000, Xiong, Jianxin wrote: > > > > > > > Heterogeneous Memory Management (HMM) utilizes > > > > > > > mmu_interval_notifier and ZONE_DEVICE to support shared > > > > > > > virtual address space and page migration between system memory > > > > > > > and device memory. HMM doesn't support pinning device memory > > > > > > > because pages located on device must be able to migrate to > > > > > > > system memory when accessed by CPU. Peer-to-peer access is > > > > > > > possible if the peer can handle page fault. For RDMA, that means the NIC must support on-demand paging. > > > > > > > > > > > > peer-peer access is currently not possible with hmm_range_fault(). > > > > > > > > > > Currently hmm_range_fault() always sets the cpu access flag and > > > > > device private pages are migrated to the system RAM in the fault handler. > > > > > However, it's possible to have a modified code flow to keep the > > > > > device private page info for use with peer to peer access. > > > > > > > > Sort of, but only within the same device, RDMA or anything else generic can't reach inside a DEVICE_PRIVATE and extract anything > > useful. > > > > > > But pfn is supposed to be all that is needed. > > > > Needed for what? The PFN of the DEVICE_PRIVATE pages is useless for anything. > > Hmm. I thought the pfn corresponds to the address in the BAR range. I could be > wrong here. No, DEVICE_PRIVATE is a dummy pfn to empty address space. Jason
