Re: [RFC PATCH 2/2] drm/amdgpu: enable gfx wave limiting for high priority compute jobs

[Christian König <christian.koenig@xxxxxxx>]

Am 28.01.21 um 17:01 schrieb Nirmoy:
> On 1/28/21 4:25 PM, Christian König wrote:
> > Am 28.01.21 um 16:21 schrieb Nirmoy:
> > > On 1/28/21 3:49 PM, Christian König wrote:
> > > > Am 28.01.21 um 15:35 schrieb Nirmoy Das:
> > > > > Enable gfx wave limiting for gfx jobs before pushing high priority
> > > > > compute jobs so that high priority compute jobs gets more resources
> > > > > to finish early.
> > > >
> > > > The problem here is what happens if you have multiple high priority 
> > > > jobs running at the same time?
> > >
> > >
> > > AFAIU, in that case quantum duration will come into effect. The 
> > > queue arbiter will switch
> > >
> > > to next high priority active queue once quantum duration expires. 
> > > This should be similar to what
> > >
> > > we already have, multiple normal priority jobs sharing GPU resources 
> > > based on CP_HQD_QUANTUM.
> > >
> > > QUEUE_DURATION register value.
> >
> > Yeah, but when the first high priority job completes it will reset 
> > mmSPI_WCL_PIPE_PERCENT_GFX back to the default value.
> >
> > Have you considered that?
>
>
> Yes I need bit of clarity here. Isn't one frame(...pm4(wave_limit), 
> pm4(IBs), pm4(restore_wave_limit), ..)
>
> executes together as one unit? If that is the case then the next high 
> prio compute job will set the wave limit
>
> again and will be applied for its dispatch call.

Yeah, that is correct. But the problem is somewhere else.

> I guess that is not the case because you asked this question. Do you 
> think we should have only one high priority
>
> queue then?

Yes exactly that. IIRC we currently have 4 low priority and 4 high 
priority queues.

The problem is those 4 high priority queues. If we only use 1 then we 
won't run into this as far as I can see.

Regards,
Christian.

> I tried to test it by running two instances of same vulkan test 
> application. I can't trace
>
> two applications together using RGP. From the trace of one 
> application(along with other running together),
>
> I didn't see any throttling down of high priority compute job(yellow 
> bars).
>
>
> Let me know what do you think. I will work with Alan to change the 
> test application so that we can verify this
>
> using multiple high priority context.
>
>
> Regards,
>
> Nirmoy
>
> > Thanks,
> > Christian.
> >
> > > Regards,
> > >
> > > Nirmoy
> > >
> > >
> > > > Christian
> > > >
> > > > > Signed-off-by: Nirmoy Das <nirmoy.das@xxxxxxx>
> > > > > ---
> > > > >   drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c | 9 +++++++++
> > > > >   1 file changed, 9 insertions(+)
> > > > >
> > > > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c 
> > > > > b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > > > > index 024d0a563a65..ee48989dfb4c 100644
> > > > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > > > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ib.c
> > > > > @@ -195,6 +195,10 @@ int amdgpu_ib_schedule(struct amdgpu_ring 
> > > > > *ring, unsigned num_ibs,
> > > > >       if ((ib->flags & AMDGPU_IB_FLAG_EMIT_MEM_SYNC) && 
> > > > > ring->funcs->emit_mem_sync)
> > > > >           ring->funcs->emit_mem_sync(ring);
> > > > >   +    if (ring->funcs->emit_wave_limit && job &&
> > > > > +        job->base.s_priority >= DRM_SCHED_PRIORITY_HIGH)
> > > > > +        ring->funcs->emit_wave_limit(ring, true);
> > > > > +
> > > > >       if (ring->funcs->insert_start)
> > > > >           ring->funcs->insert_start(ring);
> > > > >   @@ -295,6 +299,11 @@ int amdgpu_ib_schedule(struct amdgpu_ring 
> > > > > *ring, unsigned num_ibs,
> > > > >       ring->current_ctx = fence_ctx;
> > > > >       if (vm && ring->funcs->emit_switch_buffer)
> > > > >           amdgpu_ring_emit_switch_buffer(ring);
> > > > > +
> > > > > +    if (ring->funcs->emit_wave_limit && job &&
> > > > > +        job->base.s_priority >= DRM_SCHED_PRIORITY_HIGH)
> > > > > +        ring->funcs->emit_wave_limit(ring, false);
> > > > > +
> > > > >       amdgpu_ring_commit(ring);
> > > > >       return 0;
> > > > >   }
> > > >
> > > > _______________________________________________
> > > > amd-gfx mailing list
> > > > amd-gfx@xxxxxxxxxxxxxxxxxxxxx
> > > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=04%7C01%7Cnirmoy.das%40amd.com%7C67e903357ee247f9ceb008d8c3a0efdf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637474443287007930%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=EyouSUvaKjQIIWeKDilVra73iL1eb0rpnaUCDAIDvXA%3D&amp;reserved=0 
> > > _______________________________________________
> > > amd-gfx mailing list
> > > amd-gfx@xxxxxxxxxxxxxxxxxxxxx
> > > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Famd-gfx&amp;data=04%7C01%7Cnirmoy.das%40amd.com%7C67e903357ee247f9ceb008d8c3a0efdf%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637474443287007930%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=EyouSUvaKjQIIWeKDilVra73iL1eb0rpnaUCDAIDvXA%3D&amp;reserved=0 

_______________________________________________
amd-gfx mailing list
amd-gfx@xxxxxxxxxxxxxxxxxxxxx
https://lists.freedesktop.org/mailman/listinfo/amd-gfx