On Thu, 9 Jan 2025 at 17:58, Felix Kuehling <felix.kuehling@xxxxxxx> wrote:
>
> From: Christian König <christian.koenig@xxxxxxx>
>
> Try pinning into VRAM to allow P2P with RDMA NICs without ODP
> support if all attachments can do P2P. If any attachment can't do
> P2P just pin into GTT instead.
>
> Signed-off-by: Christian König <christian.koenig@xxxxxxx>
> Signed-off-by: Felix Kuehling <felix.kuehling@xxxxxxx>
> Reviewed-by: Felix Kuehling <felix.kuehling@xxxxxxx>
> Tested-by: Pak Nin Lui <pak.lui@xxxxxxx>
> Cc: Simona Vetter <simona.vetter@xxxxxxxx>
I thought the big reason we never did this was that there's no way to
account vram pinning properly, until now? But with the cgroup stuff
finally moving that's no longer a concern I guess, but I'd at least
note that down in the commit message. And maybe land this only after
the cgroup stuff is in?
Anyway code looks like how we designed this all to work, so ack.
-Sima
> ---
> drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c | 25 +++++++++++++++------
> 1 file changed, 18 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> index 8e81a83d37d84..83390143c2e9f 100644
> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
> @@ -72,11 +72,25 @@ static int amdgpu_dma_buf_attach(struct dma_buf *dmabuf,
> */
> static int amdgpu_dma_buf_pin(struct dma_buf_attachment *attach)
> {
> - struct drm_gem_object *obj = attach->dmabuf->priv;
> - struct amdgpu_bo *bo = gem_to_amdgpu_bo(obj);
> + struct dma_buf *dmabuf = attach->dmabuf;
> + struct amdgpu_bo *bo = gem_to_amdgpu_bo(dmabuf->priv);
> + u32 domains = bo->preferred_domains;
>
> - /* pin buffer into GTT */
> - return amdgpu_bo_pin(bo, AMDGPU_GEM_DOMAIN_GTT);
> + dma_resv_assert_held(dmabuf->resv);
> +
> + /*
> + * Try pinning into VRAM to allow P2P with RDMA NICs without ODP
> + * support if all attachments can do P2P. If any attachment can't do
> + * P2P just pin into GTT instead.
> + */
> + list_for_each_entry(attach, &dmabuf->attachments, node)
> + if (!attach->peer2peer)
> + domains &= ~AMDGPU_GEM_DOMAIN_VRAM;
> +
> + if (domains & AMDGPU_GEM_DOMAIN_VRAM)
> + bo->flags |= AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED;
> +
> + return amdgpu_bo_pin(bo, domains);
> }
>
> /**
> @@ -131,9 +145,6 @@ static struct sg_table *amdgpu_dma_buf_map(struct dma_buf_attachment *attach,
> r = ttm_bo_validate(&bo->tbo, &bo->placement, &ctx);
> if (r)
> return ERR_PTR(r);
> -
> - } else if (bo->tbo.resource->mem_type != TTM_PL_TT) {
> - return ERR_PTR(-EBUSY);
> }
>
> switch (bo->tbo.resource->mem_type) {
> --
> 2.34.1
>
--
Simona Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
