Fantastic, I'm integrating it now. Fingers crossed! Kent > -----Original Message----- > From: Kuehling, Felix > Sent: Tuesday, March 12, 2019 7:31 PM > To: Yang, Philip <Philip.Yang@xxxxxxx>; Russell, Kent > <Kent.Russell@xxxxxxx>; Koenig, Christian <Christian.Koenig@xxxxxxx>; > amd-gfx@xxxxxxxxxxxxxxxxxxxxx > Subject: Re: [PATCH 3/6] drm/amdgpu: allocate VM PDs/PTs on demand > > Never mind. I must have messed up my build. I can't reproduce the problem > any more. The patch I sent out is still needed and valid. AFAICT it should be all > that's needed to fix GPUVM for KFD. > > I have not seen any faults with KFDCWSRTest.BasicTest on my system with > Fiji or Vega10 with that patch applied. > > Regards, > Felix > > On 2019-03-12 5:19 p.m., Felix Kuehling wrote: > > I'm also still seeing VM faults in the eviction test even with my fix, > > and even with SDMA page table updates. There is still something else > > going wrong. :/ > > > > Thanks, > > Felix > > > > On 2019-03-12 5:13 p.m., Yang, Philip wrote: > >> vm fault happens about 1/10 for KFDCWSRTest.BasicTest for me. I am > >> using SDMA for page table update. I don't try CPU page table update. > >> > >> Philip > >> > >> On 2019-03-12 11:12 a.m., Russell, Kent wrote: > >>> Peculiar, I hit it immediately when I ran it . Can you try use > >>> --gtest_filter=KFDCWSRTest.BasicTest . That one hung every time for > me. > >>> > >>> Kent > >>> > >>>> -----Original Message----- > >>>> From: Christian König <ckoenig.leichtzumerken@xxxxxxxxx> > >>>> Sent: Tuesday, March 12, 2019 11:09 AM > >>>> To: Russell, Kent <Kent.Russell@xxxxxxx>; Koenig, Christian > >>>> <Christian.Koenig@xxxxxxx>; Kuehling, Felix > >>>> <Felix.Kuehling@xxxxxxx>; amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>> Subject: Re: [PATCH 3/6] drm/amdgpu: allocate VM PDs/PTs on > demand > >>>> > >>>> Yeah, same problem here. > >>>> > >>>> I removed libhsakmt package and installed it manually and now it > >>>> seems to work. > >>>> > >>>> Doing some testing now, but at least of hand I can't seem to > >>>> reproduce the VM fault on a Vega10. > >>>> > >>>> Christian. > >>>> > >>>> Am 12.03.19 um 16:01 schrieb Russell, Kent: > >>>>> Oh right, I remember that issue. I had that happen to me once, > >>>>> where my > >>>> installed libhsakmt didn't match up with the latest source code, so > >>>> I ended up having to remove the libhsakmt package and pointing it > >>>> to the folders instead. > >>>>> Kent > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Koenig, Christian > >>>>>> Sent: Tuesday, March 12, 2019 10:49 AM > >>>>>> To: Russell, Kent <Kent.Russell@xxxxxxx>; Kuehling, Felix > >>>>>> <Felix.Kuehling@xxxxxxx>; amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>> Subject: Re: [PATCH 3/6] drm/amdgpu: allocate VM PDs/PTs on > >>>>>> demand > >>>>>> > >>>>>> Yeah, the problem is I do have the libhsakmt installed. > >>>>>> > >>>>>> Going to give it a try to specify the directory directly. > >>>>>> > >>>>>> Christian. > >>>>>> > >>>>>> Am 12.03.19 um 15:47 schrieb Russell, Kent: > >>>>>>> The README.txt file inside the tests/kfdtest folder has > >>>>>>> instructions on how > >>>>>> to do it if you don't have the libhsakmt package installed on > >>>>>> your system: > >>>>>>> export LIBHSAKMT_PATH=/*your local libhsakmt folder*/ With > that, > >>>>>>> the headers and libraries are searched under > >>>>>>> LIBHSAKMT_PATH/include and LIBHSAKMT_PATH/lib respectively. > >>>>>>> > >>>>>>> So if you try export LIBHSAKMT_PATH as the root ROCT folder (the > >>>>>>> one > >>>>>> containing include, src, tests, etc), then that should cover it. > >>>>>>> Kent > >>>>>>> > >>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Christian König <ckoenig.leichtzumerken@xxxxxxxxx> > >>>>>>>> Sent: Tuesday, March 12, 2019 9:13 AM > >>>>>>>> To: Russell, Kent <Kent.Russell@xxxxxxx>; Kuehling, Felix > >>>>>>>> <Felix.Kuehling@xxxxxxx>; Koenig, Christian > >>>>>>>> <Christian.Koenig@xxxxxxx>; amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>>>> Subject: Re: [PATCH 3/6] drm/amdgpu: allocate VM PDs/PTs on > >>>> demand > >>>>>>>> Hi guys, > >>>>>>>> > >>>>>>>> so found a few minutes today to compile kfdtest. > >>>>>>>> > >>>>>>>> Problem is that during the compile I get a lots of this: > >>>>>>>>> CMakeFiles/kfdtest.dir/src/BaseQueue.cpp.o: In Funktion > >>>>>>>>> »BaseQueue::Create(unsigned int, unsigned int, unsigned > long*)«: > >>>>>>>>> /usr/src/ROCT-Thunk- > Interface/tests/kfdtest/src/BaseQueue.cpp:57: > >>>>>>>>> Warnung: undefinierter Verweis auf »hsaKmtCreateQueue« > >>>>>>>> Any idea? > >>>>>>>> > >>>>>>>> Christian. > >>>>>>>> > >>>>>>>> Am 11.03.19 um 17:55 schrieb Christian König: > >>>>>>>>> Hi guys, > >>>>>>>>> > >>>>>>>>> well it's most likely some missing handling in the KFD, so I'm > >>>>>>>>> rather reluctant to revert the change immediately. > >>>>>>>>> > >>>>>>>>> Problem is that I don't have time right now to look into it > >>>>>>>>> immediately. So Kent can you continue to take a look? > >>>>>>>>> > >>>>>>>>> Sounds like its crashing immediately, so it should be > >>>>>>>>> something > >>>> obvious. > >>>>>>>>> Christian. > >>>>>>>>> > >>>>>>>>> Am 11.03.19 um 10:49 schrieb Russell, Kent: > >>>>>>>>>> From what I've been able to dig through, the VM Fault > >>>>>>>>>> seems to occur right after a doorbell mmap, but that's as far > >>>>>>>>>> as I got. I can try to revert it in today's merge and see how > >>>>>>>>>> things go. > >>>>>>>>>> > >>>>>>>>>> Kent > >>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: Kuehling, Felix > >>>>>>>>>>> Sent: Friday, March 08, 2019 11:16 PM > >>>>>>>>>>> To: Koenig, Christian <Christian.Koenig@xxxxxxx>; Russell, > >>>>>>>>>>> Kent <Kent.Russell@xxxxxxx>; amd- > gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>>>>>>> Subject: RE: [PATCH 3/6] drm/amdgpu: allocate VM PDs/PTs > on > >>>>>> demand > >>>>>>>>>>> My concerns were related to eviction fence handing. It would > >>>>>>>>>>> manifest by unnecessary eviction callbacks into KFD that > >>>>>>>>>>> aren't cause by real evictions. I addressed that with a > >>>>>>>>>>> previous patch series that removed the need to remove > >>>>>>>>>>> eviction fences and add them back around page table updates > >>>>>>>>>>> in > >>>> amdgpu_amdkfd_gpuvm.c. > >>>>>>>>>>> I don't know what's going on here. I can probably take a > >>>>>>>>>>> look on Monday. I haven't considered what changed with > >>>>>>>>>>> respect to PD updates. > >>>>>>>>>>> > >>>>>>>>>>> Kent, can we temporarily revert the offending change in > >>>>>>>>>>> amd-kfd-staging just to unblock the merge? > >>>>>>>>>>> > >>>>>>>>>>> Christian, I think KFD is currently broken on > >>>>>>>>>>> amd-staging-drm-next. > >>>>>>>>>>> If we're > >>>>>>>>>>> serious about supporting KFD upstream, you may also want to > >>>>>>>>>>> consider reverting your change there for now. Also consider > >>>>>>>>>>> building the Thunk and kfdtest so you can do quick smoke > >>>>>>>>>>> tests locally whenever you make amdgpu_vm changes that > can > >>>>>>>>>>> affect > >>>> KFD. > >>>>>>>>>>> https://github.com/RadeonOpenCompute/ROCT-Thunk- > Interface > >>>>>>>>>>> > >>>>>>>>>>> Regards, > >>>>>>>>>>> Felix > >>>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: amd-gfx <amd-gfx-bounces@xxxxxxxxxxxxxxxxxxxxx> On > >>>> Behalf > >>>>>> Of > >>>>>>>>>>> Christian König > >>>>>>>>>>> Sent: Friday, March 08, 2019 9:14 AM > >>>>>>>>>>> To: Russell, Kent <Kent.Russell@xxxxxxx>; > >>>>>>>>>>> amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>>>>>>> Subject: Re: [PATCH 3/6] drm/amdgpu: allocate VM PDs/PTs > on > >>>>>> demand > >>>>>>>>>>> My best guess is that we forget somewhere to update the > PDs. > >>>>>>>>>>> What hardware is that on? > >>>>>>>>>>> > >>>>>>>>>>> Felix already mentioned that this could be problematic for > >>>>>>>>>>> the KFD. > >>>>>>>>>>> > >>>>>>>>>>> Maybe he has an idea, > >>>>>>>>>>> Christian. > >>>>>>>>>>> > >>>>>>>>>>> Am 08.03.19 um 15:04 schrieb Russell, Kent: > >>>>>>>>>>>> Hi Christian, > >>>>>>>>>>>> > >>>>>>>>>>>> This patch ended up causing a VM Fault in KFDTest. > >>>>>>>>>>>> Reverting just this > >>>>>>>>>>> patch addressed the issue: > >>>>>>>>>>>> [ 82.703503] amdgpu 0000:0c:00.0: GPU fault detected: 146 > >>>>>>>>>>>> 0x0000480c for > >>>>>>>>>>> process pid 0 thread pid 0 > >>>>>>>>>>>> [ 82.703512] amdgpu 0000:0c:00.0: > >>>>>>>>>>> VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00001000 > >>>>>>>>>>>> [ 82.703516] amdgpu 0000:0c:00.0: > >>>>>>>>>>> VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x1004800C > >>>>>>>>>>>> [ 82.703522] amdgpu 0000:0c:00.0: VM fault (0x0c, vmid 8, > >>>>>>>>>>>> pasid > >>>>>>>>>>>> 32769) at > >>>>>>>>>>> page 4096, read from 'TC0' (0x54433000) (72) > >>>>>>>>>>>> [ 82.703585] Evicting PASID 32769 queues > >>>>>>>>>>>> > >>>>>>>>>>>> I am looking into it, but if you have any insight that > >>>>>>>>>>>> would be great in > >>>>>>>>>>> helping to resolve it quickly. > >>>>>>>>>>>> Kent > >>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>> From: amd-gfx <amd-gfx-bounces@xxxxxxxxxxxxxxxxxxxxx> > On > >>>>>> Behalf > >>>>>>>> Of > >>>>>>>>>>>>> Christian König > >>>>>>>>>>>>> Sent: Tuesday, February 26, 2019 7:47 AM > >>>>>>>>>>>>> To: amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>>>>>>>>> Subject: [PATCH 3/6] drm/amdgpu: allocate VM PDs/PTs on > >>>>>> demand > >>>>>>>>>>>>> Let's start to allocate VM PDs/PTs on demand instead of > >>>>>>>>>>>>> pre-allocating them during mapping. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Signed-off-by: Christian König > <christian.koenig@xxxxxxx> > >>>>>>>>>>>>> Reviewed-by: Felix Kuehling <Felix.Kuehling@xxxxxxx> > >>>>>>>>>>>>> --- > >>>>>>>>>>>>> .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 10 > >>>> +- > >>>>>>>>>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c | 9 -- > >>>>>>>>>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 10 -- > >>>>>>>>>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 136 > >>>>>>>>>>>>> +++++------------ > >>>>>>>>>>> - > >>>>>>>>>>>>> drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 3 - > >>>>>>>>>>>>> 5 files changed, 39 insertions(+), 129 deletions(-) > >>>>>>>>>>>>> > >>>>>>>>>>>>> diff --git > >>>>>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c > >>>>>>>>>>>>> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c > >>>>>>>>>>>>> index 31e3953dcb6e..088e9b6b765b 100644 > >>>>>>>>>>>>> --- > a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c > >>>>>>>>>>>>> +++ > >>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c > >>>>>>>>>>>>> @@ -410,15 +410,7 @@ static int add_bo_to_vm(struct > >>>>>>>> amdgpu_device > >>>>>>>>>>>>> *adev, struct kgd_mem *mem, > >>>>>>>>>>>>> if (p_bo_va_entry) > >>>>>>>>>>>>> *p_bo_va_entry = bo_va_entry; > >>>>>>>>>>>>> > >>>>>>>>>>>>> - /* Allocate new page tables if needed and validate > >>>>>>>>>>>>> - * them. > >>>>>>>>>>>>> - */ > >>>>>>>>>>>>> - ret = amdgpu_vm_alloc_pts(adev, vm, va, > >>>>>>>>>>>>> amdgpu_bo_size(bo)); > >>>>>>>>>>>>> - if (ret) { > >>>>>>>>>>>>> - pr_err("Failed to allocate pts, err=%d\n", ret); > >>>>>>>>>>>>> - goto err_alloc_pts; > >>>>>>>>>>>>> - } > >>>>>>>>>>>>> - > >>>>>>>>>>>>> + /* Allocate validate page tables if needed */ > >>>>>>>>>>>>> ret = vm_validate_pt_pd_bos(vm); > >>>>>>>>>>>>> if (ret) { > >>>>>>>>>>>>> pr_err("validate_pt_pd_bos() failed\n"); diff --git > >>>>>>>>>>>>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > >>>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > >>>>>>>>>>>>> index 7e22be7ca68a..54dd02a898b9 100644 > >>>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > >>>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_csa.c > >>>>>>>>>>>>> @@ -92,15 +92,6 @@ int amdgpu_map_static_csa(struct > >>>>>>>> amdgpu_device > >>>>>>>>>>>>> *adev, struct amdgpu_vm *vm, > >>>>>>>>>>>>> return -ENOMEM; > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> - r = amdgpu_vm_alloc_pts(adev, (*bo_va)->base.vm, > >>>>>>>>>>>>> csa_addr, > >>>>>>>>>>>>> - size); > >>>>>>>>>>>>> - if (r) { > >>>>>>>>>>>>> - DRM_ERROR("failed to allocate pts for static CSA, > >>>>>>>>>>>>> err=%d\n", r); > >>>>>>>>>>>>> - amdgpu_vm_bo_rmv(adev, *bo_va); > >>>>>>>>>>>>> - ttm_eu_backoff_reservation(&ticket, &list); > >>>>>>>>>>>>> - return r; > >>>>>>>>>>>>> - } > >>>>>>>>>>>>> - > >>>>>>>>>>>>> r = amdgpu_vm_bo_map(adev, *bo_va, csa_addr, 0, > >>>>>>>>>>>>> size, AMDGPU_PTE_READABLE | > AMDGPU_PTE_WRITEABLE > >>>>>>>>>>>>> | > >>>>>>>>>>>>> AMDGPU_PTE_EXECUTABLE); diff --git > >>>>>>>>>>>>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c > >>>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c > >>>>>>>>>>>>> index 555285e329ed..fcaaac30e84b 100644 > >>>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c > >>>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c > >>>>>>>>>>>>> @@ -625,11 +625,6 @@ int amdgpu_gem_va_ioctl(struct > >>>>>> drm_device > >>>>>>>>>>> *dev, > >>>>>>>>>>>>> void *data, > >>>>>>>>>>>>> > >>>>>>>>>>>>> switch (args->operation) { > >>>>>>>>>>>>> case AMDGPU_VA_OP_MAP: > >>>>>>>>>>>>> - r = amdgpu_vm_alloc_pts(adev, bo_va->base.vm, > >>>>>>>>>>>>> args- > >>>>>>>>>>>>>> va_address, > >>>>>>>>>>>>> - args->map_size); > >>>>>>>>>>>>> - if (r) > >>>>>>>>>>>>> - goto error_backoff; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> va_flags = amdgpu_gmc_get_pte_flags(adev, > >>>>>>>>>>>>> args->flags); > >>>>>>>>>>>>> r = amdgpu_vm_bo_map(adev, bo_va, > >>>>>>>>>>>>> args->va_address, > >>>>>>>>>>>>> args->offset_in_bo, args->map_size, > >>>>>>>>>>>>> @@ > >>>>>>>>>>>>> - > >>>>>>>>>>>>> 645,11 +640,6 @@ int amdgpu_gem_va_ioctl(struct > drm_device > >>>>>> *dev, > >>>>>>>>>>> void > >>>>>>>>>>>>> *data, > >>>>>>>>>>>>> args->map_size); > >>>>>>>>>>>>> break; > >>>>>>>>>>>>> case AMDGPU_VA_OP_REPLACE: > >>>>>>>>>>>>> - r = amdgpu_vm_alloc_pts(adev, bo_va->base.vm, > >>>>>>>>>>>>> args- > >>>>>>>>>>>>>> va_address, > >>>>>>>>>>>>> - args->map_size); > >>>>>>>>>>>>> - if (r) > >>>>>>>>>>>>> - goto error_backoff; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> va_flags = amdgpu_gmc_get_pte_flags(adev, > >>>>>>>>>>>>> args->flags); > >>>>>>>>>>>>> r = amdgpu_vm_bo_replace_map(adev, bo_va, > >>>>>>>>>>>>> args- > >>>>>>>>>>>>>> va_address, > >>>>>>>>>>>>> args->offset_in_bo, args- > >>>>>>>>>>>>>> map_size, diff --git > >>>>>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c > >>>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c > >>>>>>>>>>>>> index 362436f4e856..dfad543fc000 100644 > >>>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c > >>>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c > >>>>>>>>>>>>> @@ -504,47 +504,6 @@ static void > amdgpu_vm_pt_next(struct > >>>>>>>>>>>>> amdgpu_device *adev, > >>>>>>>>>>>>> } > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> -/** > >>>>>>>>>>>>> - * amdgpu_vm_pt_first_leaf - get first leaf PD/PT > >>>>>>>>>>>>> - * > >>>>>>>>>>>>> - * @adev: amdgpu_device pointer > >>>>>>>>>>>>> - * @vm: amdgpu_vm structure > >>>>>>>>>>>>> - * @start: start addr of the walk > >>>>>>>>>>>>> - * @cursor: state to initialize > >>>>>>>>>>>>> - * > >>>>>>>>>>>>> - * Start a walk and go directly to the leaf node. > >>>>>>>>>>>>> - */ > >>>>>>>>>>>>> -static void amdgpu_vm_pt_first_leaf(struct > amdgpu_device > >>>>>> *adev, > >>>>>>>>>>>>> - struct amdgpu_vm *vm, uint64_t start, > >>>>>>>>>>>>> - struct amdgpu_vm_pt_cursor *cursor) > >>>>>>>>>>>>> -{ > >>>>>>>>>>>>> - amdgpu_vm_pt_start(adev, vm, start, cursor); > >>>>>>>>>>>>> - while (amdgpu_vm_pt_descendant(adev, cursor)); -} > >>>>>>>>>>>>> - > >>>>>>>>>>>>> -/** > >>>>>>>>>>>>> - * amdgpu_vm_pt_next_leaf - get next leaf PD/PT > >>>>>>>>>>>>> - * > >>>>>>>>>>>>> - * @adev: amdgpu_device pointer > >>>>>>>>>>>>> - * @cursor: current state > >>>>>>>>>>>>> - * > >>>>>>>>>>>>> - * Walk the PD/PT tree to the next leaf node. > >>>>>>>>>>>>> - */ > >>>>>>>>>>>>> -static void amdgpu_vm_pt_next_leaf(struct > amdgpu_device > >>>>>> *adev, > >>>>>>>>>>>>> - struct amdgpu_vm_pt_cursor *cursor) -{ > >>>>>>>>>>>>> - amdgpu_vm_pt_next(adev, cursor); > >>>>>>>>>>>>> - if (cursor->pfn != ~0ll) > >>>>>>>>>>>>> - while (amdgpu_vm_pt_descendant(adev, cursor)); -} > >>>>>>>>>>>>> - > >>>>>>>>>>>>> -/** > >>>>>>>>>>>>> - * for_each_amdgpu_vm_pt_leaf - walk over all leaf > >>>>>>>>>>>>> PDs/PTs in the hierarchy > >>>>>>>>>>>>> - */ > >>>>>>>>>>>>> -#define for_each_amdgpu_vm_pt_leaf(adev, vm, start, > end, > >>>>>>>>>>>>> cursor) > >>>>>>>>>>>>> \ > >>>>>>>>>>>>> - for (amdgpu_vm_pt_first_leaf((adev), (vm), (start), > >>>>>>>>>>>>> &(cursor)); > >>>>>>>>>>>>> \ > >>>>>>>>>>>>> - (cursor).pfn <= end; > >>>>>>>>>>>>> amdgpu_vm_pt_next_leaf((adev), > >>>>>>>>>>>>> &(cursor))) > >>>>>>>>>>>>> - > >>>>>>>>>>>>> /** > >>>>>>>>>>>>> * amdgpu_vm_pt_first_dfs - start a deep first > >>>>>>>>>>>>> search > >>>>>>>>>>>>> * > >>>>>>>>>>>>> @@ -915,74 +874,51 @@ static void > >>>> amdgpu_vm_bo_param(struct > >>>>>>>>>>>>> amdgpu_device *adev, struct amdgpu_vm *vm, > >>>>>>>>>>>>> * Returns: > >>>>>>>>>>>>> * 0 on success, errno otherwise. > >>>>>>>>>>>>> */ > >>>>>>>>>>>>> -int amdgpu_vm_alloc_pts(struct amdgpu_device *adev, > >>>>>>>>>>>>> - struct amdgpu_vm *vm, > >>>>>>>>>>>>> - uint64_t saddr, uint64_t size) > >>>>>>>>>>>>> +static int amdgpu_vm_alloc_pts(struct amdgpu_device > >>>>>>>>>>>>> +*adev, > >>>>>>>>>>>>> + struct amdgpu_vm *vm, > >>>>>>>>>>>>> + struct amdgpu_vm_pt_cursor *cursor) > >>>>>>>>>>>>> { > >>>>>>>>>>>>> - struct amdgpu_vm_pt_cursor cursor; > >>>>>>>>>>>>> + struct amdgpu_vm_pt *entry = cursor->entry; > >>>>>>>>>>>>> + struct amdgpu_bo_param bp; > >>>>>>>>>>>>> struct amdgpu_bo *pt; > >>>>>>>>>>>>> - uint64_t eaddr; > >>>>>>>>>>>>> int r; > >>>>>>>>>>>>> > >>>>>>>>>>>>> - /* validate the parameters */ > >>>>>>>>>>>>> - if (saddr & AMDGPU_GPU_PAGE_MASK || size & > >>>>>>>>>>>>> AMDGPU_GPU_PAGE_MASK) > >>>>>>>>>>>>> - return -EINVAL; > >>>>>>>>>>>>> + if (cursor->level < AMDGPU_VM_PTB && !entry- > >entries) > >>>>>>>>>>>>> +{ > >>>>>>>>>>>>> + unsigned num_entries; > >>>>>>>>>>>>> > >>>>>>>>>>>>> - eaddr = saddr + size - 1; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - saddr /= AMDGPU_GPU_PAGE_SIZE; > >>>>>>>>>>>>> - eaddr /= AMDGPU_GPU_PAGE_SIZE; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - if (eaddr >= adev->vm_manager.max_pfn) { > >>>>>>>>>>>>> - dev_err(adev->dev, "va above limit (0x%08llX >= > >>>>>>>>>>>>> 0x%08llX)\n", > >>>>>>>>>>>>> - eaddr, adev->vm_manager.max_pfn); > >>>>>>>>>>>>> - return -EINVAL; > >>>>>>>>>>>>> + num_entries = amdgpu_vm_num_entries(adev, > cursor- > >>>>>>>>>>>>>> level); > >>>>>>>>>>>>> + entry->entries = kvmalloc_array(num_entries, > >>>>>>>>>>>>> +sizeof(*entry->entries), > >>>>>>>>>>>>> + GFP_KERNEL | __GFP_ZERO); > >>>>>>>>>>>>> + if (!entry->entries) > >>>>>>>>>>>>> + return -ENOMEM; > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> - for_each_amdgpu_vm_pt_leaf(adev, vm, saddr, eaddr, > >>>>>>>>>>>>> cursor) { > >>>>>>>>>>>>> - struct amdgpu_vm_pt *entry = cursor.entry; > >>>>>>>>>>>>> - struct amdgpu_bo_param bp; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - if (cursor.level < AMDGPU_VM_PTB) { > >>>>>>>>>>>>> - unsigned num_entries; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - num_entries = amdgpu_vm_num_entries(adev, > >>>>>>>>>>>>> cursor.level); > >>>>>>>>>>>>> - entry->entries = kvmalloc_array(num_entries, > >>>>>>>>>>>>> - sizeof(*entry- > >>>>>>>>>>>>>> entries), > >>>>>>>>>>>>> - GFP_KERNEL | > >>>>>>>>>>>>> - __GFP_ZERO); > >>>>>>>>>>>>> - if (!entry->entries) > >>>>>>>>>>>>> - return -ENOMEM; > >>>>>>>>>>>>> - } > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - if (entry->base.bo) > >>>>>>>>>>>>> - continue; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - amdgpu_vm_bo_param(adev, vm, cursor.level, &bp); > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - r = amdgpu_bo_create(adev, &bp, &pt); > >>>>>>>>>>>>> - if (r) > >>>>>>>>>>>>> - return r; > >>>>>>>>>>>>> - > >>>>>>>>>>>>> - if (vm->use_cpu_for_update) { > >>>>>>>>>>>>> - r = amdgpu_bo_kmap(pt, NULL); > >>>>>>>>>>>>> - if (r) > >>>>>>>>>>>>> - goto error_free_pt; > >>>>>>>>>>>>> - } > >>>>>>>>>>>>> + if (entry->base.bo) > >>>>>>>>>>>>> + return 0; > >>>>>>>>>>>>> > >>>>>>>>>>>>> - /* Keep a reference to the root directory to > >>>>>>>>>>>>> avoid > >>>>>>>>>>>>> - * freeing them up in the wrong order. > >>>>>>>>>>>>> - */ > >>>>>>>>>>>>> - pt->parent = > >>>>>>>>>>>>> amdgpu_bo_ref(cursor.parent->base.bo); > >>>>>>>>>>>>> + amdgpu_vm_bo_param(adev, vm, cursor->level, &bp); > >>>>>>>>>>>>> > >>>>>>>>>>>>> - amdgpu_vm_bo_base_init(&entry->base, vm, pt); > >>>>>>>>>>>>> + r = amdgpu_bo_create(adev, &bp, &pt); > >>>>>>>>>>>>> + if (r) > >>>>>>>>>>>>> + return r; > >>>>>>>>>>>>> > >>>>>>>>>>>>> - r = amdgpu_vm_clear_bo(adev, vm, pt); > >>>>>>>>>>>>> + if (vm->use_cpu_for_update) { > >>>>>>>>>>>>> + r = amdgpu_bo_kmap(pt, NULL); > >>>>>>>>>>>>> if (r) > >>>>>>>>>>>>> goto error_free_pt; > >>>>>>>>>>>>> } > >>>>>>>>>>>>> > >>>>>>>>>>>>> + /* Keep a reference to the root directory to avoid > >>>>>>>>>>>>> + * freeing them up in the wrong order. > >>>>>>>>>>>>> + */ > >>>>>>>>>>>>> + pt->parent = amdgpu_bo_ref(cursor->parent- > >base.bo); > >>>>>>>>>>>>> + amdgpu_vm_bo_base_init(&entry->base, vm, pt); > >>>>>>>>>>>>> + > >>>>>>>>>>>>> + r = amdgpu_vm_clear_bo(adev, vm, pt); > >>>>>>>>>>>>> + if (r) > >>>>>>>>>>>>> + goto error_free_pt; > >>>>>>>>>>>>> + > >>>>>>>>>>>>> return 0; > >>>>>>>>>>>>> > >>>>>>>>>>>>> error_free_pt: > >>>>>>>>>>>>> @@ -1627,6 +1563,7 @@ static int > >>>>>> amdgpu_vm_update_ptes(struct > >>>>>>>>>>>>> amdgpu_pte_update_params *params, > >>>>>>>>>>>>> struct amdgpu_vm_pt_cursor cursor; > >>>>>>>>>>>>> uint64_t frag_start = start, frag_end; > >>>>>>>>>>>>> unsigned int frag; > >>>>>>>>>>>>> + int r; > >>>>>>>>>>>>> > >>>>>>>>>>>>> /* figure out the initial fragment */ > >>>>>>>>>>>>> amdgpu_vm_fragment(params, frag_start, end, > >>>>>>>>>>>>> flags, &frag, &frag_end); @@ -1634,12 +1571,15 @@ static > >>>>>>>>>>>>> int amdgpu_vm_update_ptes(struct > amdgpu_pte_update_params > >>>>>>>> *params, > >>>>>>>>>>>>> /* walk over the address space and update the > >>>>>>>>>>>>> PTs */ > >>>>>>>>>>>>> amdgpu_vm_pt_start(adev, params->vm, start, > >>>>>>>>>>>>> &cursor); > >>>>>>>>>>>>> while (cursor.pfn < end) { > >>>>>>>>>>>>> - struct amdgpu_bo *pt = cursor.entry->base.bo; > >>>>>>>>>>>>> unsigned shift, parent_shift, mask; > >>>>>>>>>>>>> uint64_t incr, entry_end, pe_start; > >>>>>>>>>>>>> + struct amdgpu_bo *pt; > >>>>>>>>>>>>> > >>>>>>>>>>>>> - if (!pt) > >>>>>>>>>>>>> - return -ENOENT; > >>>>>>>>>>>>> + r = amdgpu_vm_alloc_pts(params->adev, params- > >vm, > >>>>>>>>>>>>> &cursor); > >>>>>>>>>>>>> + if (r) > >>>>>>>>>>>>> + return r; > >>>>>>>>>>>>> + > >>>>>>>>>>>>> + pt = cursor.entry->base.bo; > >>>>>>>>>>>>> > >>>>>>>>>>>>> /* The root level can't be a huge page */ > >>>>>>>>>>>>> if (cursor.level == > >>>>>>>>>>>>> adev->vm_manager.root_level) > >>>>>>>>>>>>> { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h > >>>>>>>>>>>>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h > >>>>>>>>>>>>> index 81ff8177f092..116605c038d2 100644 > >>>>>>>>>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h > >>>>>>>>>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h > >>>>>>>>>>>>> @@ -303,9 +303,6 @@ bool amdgpu_vm_ready(struct > >>>> amdgpu_vm > >>>>>>>> *vm); > >>>>>>>>>>> int > >>>>>>>>>>>>> amdgpu_vm_validate_pt_bos(struct amdgpu_device > *adev, > >>>> struct > >>>>>>>>>>>>> amdgpu_vm *vm, > >>>>>>>>>>>>> int (*callback)(void *p, struct > >>>>>>>>>>>>> amdgpu_bo *bo), > >>>>>>>>>>>>> void *param); -int > >>>>>>>>>>>>> amdgpu_vm_alloc_pts(struct amdgpu_device *adev, > >>>>>>>>>>>>> - struct amdgpu_vm *vm, > >>>>>>>>>>>>> - uint64_t saddr, uint64_t size); > >>>>>>>>>>>>> int amdgpu_vm_flush(struct amdgpu_ring *ring, > >>>>>>>>>>>>> struct amdgpu_job *job, bool need_pipe_sync); int > >>>>>>>>>>>>> amdgpu_vm_update_directories(struct > >>>>>>>>>>>>> amdgpu_device *adev, > >>>>>>>>>>>>> struct amdgpu_vm *vm); > >>>>>>>>>>>>> -- > >>>>>>>>>>>>> 2.17.1 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > _______________________________________________ > >>>>>>>>>>>>> amd-gfx mailing list > >>>>>>>>>>>>> amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>>>>>>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> amd-gfx mailing list > >>>>>>>>>>> amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>>>>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> amd-gfx mailing list > >>>>>>>>>> amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>>>>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx > >>>>> _______________________________________________ > >>>>> amd-gfx mailing list > >>>>> amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>>>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx > >>> _______________________________________________ > >>> amd-gfx mailing list > >>> amd-gfx@xxxxxxxxxxxxxxxxxxxxx > >>> https://lists.freedesktop.org/mailman/listinfo/amd-gfx > >>> _______________________________________________ amd-gfx mailing list amd-gfx@xxxxxxxxxxxxxxxxxxxxx https://lists.freedesktop.org/mailman/listinfo/amd-gfx
