Re: Preparation for unpinned DMA-buf handling

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, May 15, 2019 at 11:24 PM Alex Deucher <alexdeucher@xxxxxxxxx> wrote:
>
> On Wed, May 15, 2019 at 4:52 PM Daniel Vetter <daniel@xxxxxxxx> wrote:
> >
> > On Wed, May 15, 2019 at 10:08 PM Alex Deucher <alexdeucher@xxxxxxxxx> wrote:
> > >
> > > On Wed, May 15, 2019 at 4:54 AM Daniel Vetter <daniel@xxxxxxxx> wrote:
> > > >
> > > > On Tue, May 14, 2019 at 04:05:13PM +0200, Christian König wrote:
> > > > > Hi Daniel,
> > > > >
> > > > > since we are not moving forward with this I've separated the change out of the larger patchset.
> > > > >
> > > > > Any objections to this now? It's basically a 1:1 move of the functionality from DRM to DMA-buf.
> > > >
> > > > For my big picture understanding, and because I spotted in the latest rocm
> > > > release that apparently the fancy interconnect is now support. I looked
> > > > around a bit in upstream and basic xgmi support seems to be there already,
> > > > but I haven't seen anything that looks like xgmi buffer sharing is there
> > > > already? Missing something, or am I looking at the wrong thing (xgmi seems
> > > > to be the engineering name for that fancy interconnect at least).
> > > >
> > > > For the big plan I've seen the dynamic dma-buf and p2p stuff, I think it'd
> > > > be good if we also make sure xgmi and related things integrate somewhat
> > > > neatly. Ofc xgmi requires that we upcast the dma_buf_attachment to
> > > > somethign driver specific (I don't think we should make that generic, but
> > > > maybe Jerome has different ideas with HMM). Question I have is how much of
> > > > the map/unmap/invaliidate and passing sg tables around we can reuse for
> > > > this.
> > >
> > > xgmi is an optimization in the driver.  If a dma-buf comes in, we'd
> > > check the drm-buf pointers to verify that it is an amdgpu buffer, then
> > > we'd check if the other GPU is connected via xgmi or not.  If so, we'd
> > > use the internal xgmi address when setting up our GPUVM address space,
> > > otherwise, we'd use the dma address.
> > >
> > > The code to actually enable sharing over xgmi (or p2p for that matter)
> > > is not upstream yet.  It's pretty trivial though once the
> > > prerequisites are in place.
> > >
> > > Here's the patch to enable xgmi rather than p2p when it's available:
> > > https://cgit.freedesktop.org/~agd5f/linux/commit/?h=amd-19.10&id=7a443319db2e614114b119015daa4e2c1ed9474d
> >
> > Hm I think I looked at this, but didn't realize how it all fits. I'll
> > grab the rocm kernel tree again and browse a bit more.
> >
> > > And the original patch to add p2p support is mixed into this giant
> > > squashed commit (I'll see if I can dig up the original patch):
> > > https://cgit.freedesktop.org/~agd5f/linux/commit/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c?h=amd-19.10&id=67f5976ee4842d01af79144a8355a040ed36b6d2
> > > The relevant change is this hunk here:
> > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > > index d4d3424..49dd501 100644
> > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
> > > @@ -1674,9 +1677,15 @@ int amdgpu_vm_bo_update(struct amdgpu_device *adev,
> > >          exclusive = reservation_object_get_excl(bo->tbo.resv);
> > >      }
> > >
> > > -    if (bo)
> > > +    if (bo) {
> > >          flags = amdgpu_ttm_tt_pte_flags(adev, bo->tbo.ttm, mem);
> > > -    else
> > > +        bo_adev = amdgpu_ttm_adev(bo->tbo.bdev);
> > > +        if (mem && mem->mem_type == TTM_PL_VRAM &&
> > > +            adev != bo_adev) {
> > > +            flags |= AMDGPU_PTE_SYSTEM;
> > > +            vram_base_offset = bo_adev->gmc.aper_base;
> > > +        }
> > > +    } else
> > >          flags = 0x0;
> > >
> > >      if (clear || (bo && bo->tbo.resv == vm->root.base.bo->tbo.resv))
> > >
> > > We just treat the remote BAR like we would for system memory mapped
> > > into the GPUVM.  This will obviously need some updating to properly
> > > deal with IOMMU, etc. once dma-buf lands.
> >
> > Yeah the base address adjusting to pick the right window looks as
> > expected. I think how you get at the list of offsets for all the pages
> > of a bo is the more interesting thing. For p2p or system memory you
> > get an sg table with dma addresses (which you then need to correct for
> > where that is in gpu pa and stuff that into the gpu vm ptes ofc). For
> > the direct access/xgmi case I can see a bunch of options, one of them
> > would be to forgo the import and just directly upcast to the other
> > driver's amdgpu_bo and access everything directly. That seems to be
> > what you're doing here. Other extreme would be to somehow augment the
>
> Right.  For xgmi, all connected gpus can access the entire vram
> aperture from all connected gpus via it's local vram aperture like a
> giant vram NUMA array.  Well, assuming all driver instances have given
> permission to allow access from their xgmi peers.  At the moment we
> always do.
>
> For dma-buf, we can check and upcast based on the dma-buf pointers.
> For ROCm, we have a single device node for all GPUs in the system and
> all allocations go through that and we can use that get back to the
> right amdgpu driver instance.

Hm right the amdkfd model makes things a bit easier in this regard,
and since you have ttm locking going across device boundaries isn't a
bad idea either.
-Daniel

>
> Alex
>
> > sg table to be able to handle xgmi addresses I think.
> >
> > Not sure where we should land on this spectrum for upstream.
> > -Daniel
> >
> > >
> > > Alex
> > >
> > > >
> > > > If you have patches/pointers would be really intersting to read them a
> > > > bit.
> > > > -Daniel
> > > > --
> > > > Daniel Vetter
> > > > Software Engineer, Intel Corporation
> > > > http://blog.ffwll.ch
> > > > _______________________________________________
> > > > dri-devel mailing list
> > > > dri-devel@xxxxxxxxxxxxxxxxxxxxx
> > > > https://lists.freedesktop.org/mailman/listinfo/dri-devel
> >
> >
> >
> > --
> > Daniel Vetter
> > Software Engineer, Intel Corporation
> > +41 (0) 79 365 57 48 - http://blog.ffwll.ch



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/dri-devel




[Index of Archives]     [Linux DRI Users]     [Linux Intel Graphics]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]     [XFree86]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Linux Kernel]     [Linux SCSI]     [XFree86]
  Powered by Linux