On Fri, Aug 23, 2024 at 10:23:48AM +0100, Connor Abbott wrote: > On Fri, Aug 23, 2024 at 10:21 AM Connor Abbott <cwabbott0@xxxxxxxxx> wrote: > > > > On Thu, Aug 22, 2024 at 9:06 PM Akhil P Oommen <quic_akhilpo@xxxxxxxxxxx> wrote: > > > > > > On Wed, Aug 21, 2024 at 05:02:56PM +0100, Connor Abbott wrote: > > > > On Mon, Aug 19, 2024 at 9:09 PM Akhil P Oommen <quic_akhilpo@xxxxxxxxxxx> wrote: > > > > > > > > > > On Thu, Aug 15, 2024 at 08:26:14PM +0200, Antonino Maniscalco wrote: > > > > > > This patch implements preemption feature for A6xx targets, this allows > > > > > > the GPU to switch to a higher priority ringbuffer if one is ready. A6XX > > > > > > hardware as such supports multiple levels of preemption granularities, > > > > > > ranging from coarse grained(ringbuffer level) to a more fine grained > > > > > > such as draw-call level or a bin boundary level preemption. This patch > > > > > > enables the basic preemption level, with more fine grained preemption > > > > > > support to follow. > > > > > > > > > > > > Signed-off-by: Sharat Masetty <smasetty@xxxxxxxxxxxxxx> > > > > > > Signed-off-by: Antonino Maniscalco <antomani103@xxxxxxxxx> > > > > > > --- > > > > > > > > > > No postamble packets which resets perfcounters? It is necessary. Also, I > > > > > think we should disable preemption during profiling like we disable slumber. > > > > > > > > > > -Akhil. > > > > > > > > I don't see anything in kgsl which disables preemption during > > > > profiling. It disables resetting perfcounters when doing system-wide > > > > profiling, like freedreno, and in that case I assume preempting is > > > > fine because the system profiler has a complete view of everything and > > > > should "see" preemptions through the traces. For something like > > > > VK_KHR_performance_query I suppose we'd want to disable preemption > > > > because we disable saving/restoring perf counters, but that has to > > > > happen in userspace because the kernel doesn't know what userspace > > > > does. > > > > > > > > > > KGSL does some sort of arbitration of perfcounter configurations and > > > adds the select/enablement reg configuration as part of dynamic > > > power up register list which we are not doing here. Is this something > > > you are taking care of from userspace via preamble? > > > > > > -Akhil > > > > I don't think we have to take care of that in userspace, because Mesa > > will always configure the counter registers before reading them in the > > same submission, and if it gets preempted in the meantime then we're > > toast anyways (due to not saving/restoring perf counters). kgsl sets > > them from userspace, which is why it has to do something to set them > > Sorry, should be "kgsl sets them from the kernel". > > > after IFPC slumber or a context switch when the HW state is gone. > > Also, because the upstream approach doesn't play nicely with system > > profilers like perfetto, VK_KHR_performance_query is hidden by default > > behind a debug flag in turnip. So there's already an element of "this > > is unsupported, you have to know what you're doing to use it." But when you have composition on GPU enabled, there will be very frequent preemption. And I don't know how usable profiling tools will be in that case unless you disable preemption with a Mesa debug flag. But for that to work, all existing submitqueues should be destroyed and recreated. So I was thinking that we can use the sysprof propertry to force L0 preemption from kernel. -Akhil. > > > > Connor > > > > > > > > > Connor > > > > > > > > > > > > > > > drivers/gpu/drm/msm/Makefile | 1 + > > > > > > drivers/gpu/drm/msm/adreno/a6xx_gpu.c | 323 +++++++++++++++++++++- > > > > > > drivers/gpu/drm/msm/adreno/a6xx_gpu.h | 168 ++++++++++++ > > > > > > drivers/gpu/drm/msm/adreno/a6xx_preempt.c | 434 ++++++++++++++++++++++++++++++ > > > > > > drivers/gpu/drm/msm/msm_ringbuffer.h | 7 + > > > > > > 5 files changed, 924 insertions(+), 9 deletions(-) > > > > > > > > > > > > diff --git a/drivers/gpu/drm/msm/Makefile b/drivers/gpu/drm/msm/Makefile > > > > > > index f5e2838c6a76..32e915109a59 100644 > > > > > > --- a/drivers/gpu/drm/msm/Makefile > > > > > > +++ b/drivers/gpu/drm/msm/Makefile > > > > > > @@ -23,6 +23,7 @@ adreno-y := \ > > > > > > adreno/a6xx_gpu.o \ > > > > > > adreno/a6xx_gmu.o \ > > > > > > adreno/a6xx_hfi.o \ > > > > > > + adreno/a6xx_preempt.o \ > > > > > > > > > > > > adreno-$(CONFIG_DEBUG_FS) += adreno/a5xx_debugfs.o \ > > > > > > > > > > > > diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c > > > > > > index 32a4faa93d7f..1a90db5759b8 100644 > > > > > > --- a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c > > > > > > +++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c > > > > > > @@ -16,6 +16,83 @@ > > > > > > > > > > > > #define GPU_PAS_ID 13 > > > > > > > > > > > > +/* IFPC & Preemption static powerup restore list */ > > > > > > +static const uint32_t a7xx_pwrup_reglist[] = { > > > > > > + REG_A6XX_UCHE_TRAP_BASE, > > > > > > + REG_A6XX_UCHE_TRAP_BASE + 1, > > > > > > + REG_A6XX_UCHE_WRITE_THRU_BASE, > > > > > > + REG_A6XX_UCHE_WRITE_THRU_BASE + 1, > > > > > > + REG_A6XX_UCHE_GMEM_RANGE_MIN, > > > > > > + REG_A6XX_UCHE_GMEM_RANGE_MIN + 1, > > > > > > + REG_A6XX_UCHE_GMEM_RANGE_MAX, > > > > > > + REG_A6XX_UCHE_GMEM_RANGE_MAX + 1, > > > > > > + REG_A6XX_UCHE_CACHE_WAYS, > > > > > > + REG_A6XX_UCHE_MODE_CNTL, > > > > > > + REG_A6XX_RB_NC_MODE_CNTL, > > > > > > + REG_A6XX_RB_CMP_DBG_ECO_CNTL, > > > > > > + REG_A7XX_GRAS_NC_MODE_CNTL, > > > > > > + REG_A6XX_RB_CONTEXT_SWITCH_GMEM_SAVE_RESTORE, > > > > > > + REG_A6XX_UCHE_GBIF_GX_CONFIG, > > > > > > + REG_A6XX_UCHE_CLIENT_PF, > > > > > > +}; > > > > > > + > > > > > > +static const uint32_t a7xx_ifpc_pwrup_reglist[] = { > > > > > > + REG_A6XX_TPL1_NC_MODE_CNTL, > > > > > > + REG_A6XX_SP_NC_MODE_CNTL, > > > > > > + REG_A6XX_CP_DBG_ECO_CNTL, > > > > > > + REG_A6XX_CP_PROTECT_CNTL, > > > > > > + REG_A6XX_CP_PROTECT(0), > > > > > > + REG_A6XX_CP_PROTECT(1), > > > > > > + REG_A6XX_CP_PROTECT(2), > > > > > > + REG_A6XX_CP_PROTECT(3), > > > > > > + REG_A6XX_CP_PROTECT(4), > > > > > > + REG_A6XX_CP_PROTECT(5), > > > > > > + REG_A6XX_CP_PROTECT(6), > > > > > > + REG_A6XX_CP_PROTECT(7), > > > > > > + REG_A6XX_CP_PROTECT(8), > > > > > > + REG_A6XX_CP_PROTECT(9), > > > > > > + REG_A6XX_CP_PROTECT(10), > > > > > > + REG_A6XX_CP_PROTECT(11), > > > > > > + REG_A6XX_CP_PROTECT(12), > > > > > > + REG_A6XX_CP_PROTECT(13), > > > > > > + REG_A6XX_CP_PROTECT(14), > > > > > > + REG_A6XX_CP_PROTECT(15), > > > > > > + REG_A6XX_CP_PROTECT(16), > > > > > > + REG_A6XX_CP_PROTECT(17), > > > > > > + REG_A6XX_CP_PROTECT(18), > > > > > > + REG_A6XX_CP_PROTECT(19), > > > > > > + REG_A6XX_CP_PROTECT(20), > > > > > > + REG_A6XX_CP_PROTECT(21), > > > > > > + REG_A6XX_CP_PROTECT(22), > > > > > > + REG_A6XX_CP_PROTECT(23), > > > > > > + REG_A6XX_CP_PROTECT(24), > > > > > > + REG_A6XX_CP_PROTECT(25), > > > > > > + REG_A6XX_CP_PROTECT(26), > > > > > > + REG_A6XX_CP_PROTECT(27), > > > > > > + REG_A6XX_CP_PROTECT(28), > > > > > > + REG_A6XX_CP_PROTECT(29), > > > > > > + REG_A6XX_CP_PROTECT(30), > > > > > > + REG_A6XX_CP_PROTECT(31), > > > > > > + REG_A6XX_CP_PROTECT(32), > > > > > > + REG_A6XX_CP_PROTECT(33), > > > > > > + REG_A6XX_CP_PROTECT(34), > > > > > > + REG_A6XX_CP_PROTECT(35), > > > > > > + REG_A6XX_CP_PROTECT(36), > > > > > > + REG_A6XX_CP_PROTECT(37), > > > > > > + REG_A6XX_CP_PROTECT(38), > > > > > > + REG_A6XX_CP_PROTECT(39), > > > > > > + REG_A6XX_CP_PROTECT(40), > > > > > > + REG_A6XX_CP_PROTECT(41), > > > > > > + REG_A6XX_CP_PROTECT(42), > > > > > > + REG_A6XX_CP_PROTECT(43), > > > > > > + REG_A6XX_CP_PROTECT(44), > > > > > > + REG_A6XX_CP_PROTECT(45), > > > > > > + REG_A6XX_CP_PROTECT(46), > > > > > > + REG_A6XX_CP_PROTECT(47), > > > > > > + REG_A6XX_CP_AHB_CNTL, > > > > > > +}; > > > > > > + > > > > > > + > > > > > > static inline bool _a6xx_check_idle(struct msm_gpu *gpu) > > > > > > { > > > > > > struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > @@ -68,6 +145,8 @@ static void update_shadow_rptr(struct msm_gpu *gpu, struct msm_ringbuffer *ring) > > > > > > > > > > > > static void a6xx_flush(struct msm_gpu *gpu, struct msm_ringbuffer *ring) > > > > > > { > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > uint32_t wptr; > > > > > > unsigned long flags; > > > > > > > > > > > > @@ -81,12 +160,26 @@ static void a6xx_flush(struct msm_gpu *gpu, struct msm_ringbuffer *ring) > > > > > > /* Make sure to wrap wptr if we need to */ > > > > > > wptr = get_wptr(ring); > > > > > > > > > > > > - spin_unlock_irqrestore(&ring->preempt_lock, flags); > > > > > > - > > > > > > /* Make sure everything is posted before making a decision */ > > > > > > mb(); > > > > > > > > > > > > - gpu_write(gpu, REG_A6XX_CP_RB_WPTR, wptr); > > > > > > + /* Update HW if this is the current ring and we are not in preempt*/ > > > > > > + if (!a6xx_in_preempt(a6xx_gpu)) { > > > > > > + /* > > > > > > + * Order the reads of the preempt state and cur_ring. This > > > > > > + * matches the barrier after writing cur_ring. > > > > > > + */ > > > > > > + rmb(); > > > > > > + > > > > > > + if (a6xx_gpu->cur_ring == ring) > > > > > > + gpu_write(gpu, REG_A6XX_CP_RB_WPTR, wptr); > > > > > > + else > > > > > > + ring->skip_inline_wptr = true; > > > > > > + } else { > > > > > > + ring->skip_inline_wptr = true; > > > > > > + } > > > > > > + > > > > > > + spin_unlock_irqrestore(&ring->preempt_lock, flags); > > > > > > } > > > > > > > > > > > > static void get_stats_counter(struct msm_ringbuffer *ring, u32 counter, > > > > > > @@ -138,12 +231,14 @@ static void a6xx_set_pagetable(struct a6xx_gpu *a6xx_gpu, > > > > > > > > > > > > /* > > > > > > * Write the new TTBR0 to the memstore. This is good for debugging. > > > > > > + * Needed for preemption > > > > > > */ > > > > > > - OUT_PKT7(ring, CP_MEM_WRITE, 4); > > > > > > + OUT_PKT7(ring, CP_MEM_WRITE, 5); > > > > > > OUT_RING(ring, CP_MEM_WRITE_0_ADDR_LO(lower_32_bits(memptr))); > > > > > > OUT_RING(ring, CP_MEM_WRITE_1_ADDR_HI(upper_32_bits(memptr))); > > > > > > OUT_RING(ring, lower_32_bits(ttbr)); > > > > > > - OUT_RING(ring, (asid << 16) | upper_32_bits(ttbr)); > > > > > > + OUT_RING(ring, upper_32_bits(ttbr)); > > > > > > + OUT_RING(ring, ctx->seqno); > > > > > > > > > > > > /* > > > > > > * Sync both threads after switching pagetables and enable BR only > > > > > > @@ -268,12 +363,55 @@ static void a6xx_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) > > > > > > a6xx_flush(gpu, ring); > > > > > > } > > > > > > > > > > > > +static void a6xx_emit_set_pseudo_reg(struct msm_ringbuffer *ring, > > > > > > + struct a6xx_gpu *a6xx_gpu, struct msm_gpu_submitqueue *queue) > > > > > > +{ > > > > > > + u64 preempt_offset_priv_secure; > > > > > > + > > > > > > + OUT_PKT7(ring, CP_SET_PSEUDO_REG, 15); > > > > > > + > > > > > > + OUT_RING(ring, SMMU_INFO); > > > > > > + /* don't save SMMU, we write the record from the kernel instead */ > > > > > > + OUT_RING(ring, 0); > > > > > > + OUT_RING(ring, 0); > > > > > > + > > > > > > + /* privileged and non secure buffer save */ > > > > > > + OUT_RING(ring, NON_SECURE_SAVE_ADDR); > > > > > > + OUT_RING(ring, lower_32_bits( > > > > > > + a6xx_gpu->preempt_iova[ring->id] + PREEMPT_OFFSET_PRIV_NON_SECURE)); > > > > > > + OUT_RING(ring, upper_32_bits( > > > > > > + a6xx_gpu->preempt_iova[ring->id] + PREEMPT_OFFSET_PRIV_NON_SECURE)); > > > > > > + OUT_RING(ring, SECURE_SAVE_ADDR); > > > > > > + preempt_offset_priv_secure = > > > > > > + PREEMPT_OFFSET_PRIV_SECURE(a6xx_gpu->base.info->preempt_record_size); > > > > > > + OUT_RING(ring, lower_32_bits( > > > > > > + a6xx_gpu->preempt_iova[ring->id] + preempt_offset_priv_secure)); > > > > > > + OUT_RING(ring, upper_32_bits( > > > > > > + a6xx_gpu->preempt_iova[ring->id] + preempt_offset_priv_secure)); > > > > > > + > > > > > > + /* user context buffer save */ > > > > > > + OUT_RING(ring, NON_PRIV_SAVE_ADDR); > > > > > > + if (queue) { > > > > > > + OUT_RING(ring, lower_32_bits(queue->bo_iova)); > > > > > > + OUT_RING(ring, upper_32_bits(queue->bo_iova)); > > > > > > + } else { > > > > > > + OUT_RING(ring, 0); > > > > > > + OUT_RING(ring, 0); > > > > > > + } > > > > > > + > > > > > > + OUT_RING(ring, COUNTER); > > > > > > + /* seems OK to set to 0 to disable it */ > > > > > > + OUT_RING(ring, 0); > > > > > > + OUT_RING(ring, 0); > > > > > > +} > > > > > > + > > > > > > static void a7xx_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) > > > > > > { > > > > > > unsigned int index = submit->seqno % MSM_GPU_SUBMIT_STATS_COUNT; > > > > > > struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > struct msm_ringbuffer *ring = submit->ring; > > > > > > + uint64_t scratch_dest = SCRATCH_USER_CTX_IOVA(ring->id, a6xx_gpu); > > > > > > unsigned int i, ibs = 0; > > > > > > > > > > > > /* > > > > > > @@ -283,6 +421,25 @@ static void a7xx_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) > > > > > > OUT_PKT7(ring, CP_THREAD_CONTROL, 1); > > > > > > OUT_RING(ring, CP_THREAD_CONTROL_0_SYNC_THREADS | CP_SET_THREAD_BR); > > > > > > > > > > > > + /* > > > > > > + * If preemption is enabled, then set the pseudo register for the save > > > > > > + * sequence > > > > > > + */ > > > > > > + if (gpu->nr_rings > 1) { > > > > > > + a6xx_emit_set_pseudo_reg(ring, a6xx_gpu, submit->queue); > > > > > > + > > > > > > + /* > > > > > > + * Ask CP to save the user context buffer's iova address to a > > > > > > + * scratch memory region, this is needed if the CP preempts > > > > > > + * this ring in between this submit's IB list. > > > > > > + */ > > > > > > + OUT_PKT7(ring, CP_MEM_WRITE, 4); > > > > > > + OUT_RING(ring, lower_32_bits(scratch_dest)); > > > > > > + OUT_RING(ring, upper_32_bits(scratch_dest)); > > > > > > + OUT_RING(ring, lower_32_bits(submit->queue->bo_iova)); > > > > > > + OUT_RING(ring, upper_32_bits(submit->queue->bo_iova)); > > > > > > + } > > > > > > + > > > > > > a6xx_set_pagetable(a6xx_gpu, ring, submit->queue->ctx); > > > > > > > > > > > > get_stats_counter(ring, REG_A7XX_RBBM_PERFCTR_CP(0), > > > > > > @@ -376,6 +533,8 @@ static void a7xx_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) > > > > > > OUT_RING(ring, upper_32_bits(rbmemptr(ring, bv_fence))); > > > > > > OUT_RING(ring, submit->seqno); > > > > > > > > > > > > + a6xx_gpu->last_seqno[ring->id] = submit->seqno; > > > > > > + > > > > > > /* write the ringbuffer timestamp */ > > > > > > OUT_PKT7(ring, CP_EVENT_WRITE, 4); > > > > > > OUT_RING(ring, CACHE_CLEAN | CP_EVENT_WRITE_0_IRQ | BIT(27)); > > > > > > @@ -389,10 +548,42 @@ static void a7xx_submit(struct msm_gpu *gpu, struct msm_gem_submit *submit) > > > > > > OUT_PKT7(ring, CP_SET_MARKER, 1); > > > > > > OUT_RING(ring, 0x100); /* IFPC enable */ > > > > > > > > > > > > + /* If preemption is enabled */ > > > > > > + if (gpu->nr_rings > 1) { > > > > > > + /* > > > > > > + * Reset the scratch region as we are done with the > > > > > > + * IB list of this submission > > > > > > + */ > > > > > > + OUT_PKT7(ring, CP_MEM_WRITE, 4); > > > > > > + OUT_RING(ring, lower_32_bits(scratch_dest)); > > > > > > + OUT_RING(ring, upper_32_bits(scratch_dest)); > > > > > > + OUT_RING(ring, 0x00); > > > > > > + OUT_RING(ring, 0x00); > > > > > > + > > > > > > + /* Yield the floor on command completion */ > > > > > > + OUT_PKT7(ring, CP_CONTEXT_SWITCH_YIELD, 4); > > > > > > + > > > > > > + /* > > > > > > + * If dword[2:1] are non zero, they specify an address for > > > > > > + * the CP to write the value of dword[3] to on preemption > > > > > > + * complete. Write 0 to skip the write > > > > > > + */ > > > > > > + OUT_RING(ring, 0x00); > > > > > > + OUT_RING(ring, 0x00); > > > > > > + /* Data value - not used if the address above is 0 */ > > > > > > + OUT_RING(ring, 0x01); > > > > > > + /* generate interrupt on preemption completion */ > > > > > > + OUT_RING(ring, 0x00); > > > > > > + } > > > > > > + > > > > > > + > > > > > > trace_msm_gpu_submit_flush(submit, > > > > > > gpu_read64(gpu, REG_A6XX_CP_ALWAYS_ON_COUNTER)); > > > > > > > > > > > > a6xx_flush(gpu, ring); > > > > > > + > > > > > > + /* Check to see if we need to start preemption */ > > > > > > + a6xx_preempt_trigger(gpu); > > > > > > } > > > > > > > > > > > > static void a6xx_set_hwcg(struct msm_gpu *gpu, bool state) > > > > > > @@ -588,6 +779,89 @@ static void a6xx_set_ubwc_config(struct msm_gpu *gpu) > > > > > > adreno_gpu->ubwc_config.min_acc_len << 23 | hbb_lo << 21); > > > > > > } > > > > > > > > > > > > +static void a7xx_patch_pwrup_reglist(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + struct adreno_reglist_list reglist[2]; > > > > > > + void *ptr = a6xx_gpu->pwrup_reglist_ptr; > > > > > > + struct cpu_gpu_lock *lock = ptr; > > > > > > + u32 *dest = (u32 *)&lock->regs[0]; > > > > > > + int i, j; > > > > > > + > > > > > > + lock->gpu_req = lock->cpu_req = lock->turn = 0; > > > > > > + lock->ifpc_list_len = ARRAY_SIZE(a7xx_ifpc_pwrup_reglist); > > > > > > + lock->preemption_list_len = ARRAY_SIZE(a7xx_pwrup_reglist); > > > > > > + > > > > > > + /* Static IFPC-only registers */ > > > > > > + reglist[0].regs = a7xx_ifpc_pwrup_reglist; > > > > > > + reglist[0].count = ARRAY_SIZE(a7xx_ifpc_pwrup_reglist); > > > > > > + lock->ifpc_list_len = reglist[0].count; > > > > > > + > > > > > > + /* Static IFPC + preemption registers */ > > > > > > + reglist[1].regs = a7xx_pwrup_reglist; > > > > > > + reglist[1].count = ARRAY_SIZE(a7xx_pwrup_reglist); > > > > > > + lock->preemption_list_len = reglist[1].count; > > > > > > + > > > > > > + /* > > > > > > + * For each entry in each of the lists, write the offset and the current > > > > > > + * register value into the GPU buffer > > > > > > + */ > > > > > > + for (i = 0; i < 2; i++) { > > > > > > + const u32 *r = reglist[i].regs; > > > > > > + > > > > > > + for (j = 0; j < reglist[i].count; j++) { > > > > > > + *dest++ = r[j]; > > > > > > + *dest++ = gpu_read(gpu, r[j]); > > > > > > + } > > > > > > + } > > > > > > + > > > > > > + /* > > > > > > + * The overall register list is composed of > > > > > > + * 1. Static IFPC-only registers > > > > > > + * 2. Static IFPC + preemption registers > > > > > > + * 3. Dynamic IFPC + preemption registers (ex: perfcounter selects) > > > > > > + * > > > > > > + * The first two lists are static. Size of these lists are stored as > > > > > > + * number of pairs in ifpc_list_len and preemption_list_len > > > > > > + * respectively. With concurrent binning, Some of the perfcounter > > > > > > + * registers being virtualized, CP needs to know the pipe id to program > > > > > > + * the aperture inorder to restore the same. Thus, third list is a > > > > > > + * dynamic list with triplets as > > > > > > + * (<aperture, shifted 12 bits> <address> <data>), and the length is > > > > > > + * stored as number for triplets in dynamic_list_len. > > > > > > + */ > > > > > > + lock->dynamic_list_len = 0; > > > > > > +} > > > > > > + > > > > > > +static int a7xx_preempt_start(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + struct msm_ringbuffer *ring = gpu->rb[0]; > > > > > > + > > > > > > + if (gpu->nr_rings <= 1) > > > > > > + return 0; > > > > > > + > > > > > > + /* Turn CP protection off */ > > > > > > + OUT_PKT7(ring, CP_SET_PROTECTED_MODE, 1); > > > > > > + OUT_RING(ring, 0); > > > > > > + > > > > > > + a6xx_emit_set_pseudo_reg(ring, a6xx_gpu, NULL); > > > > > > + > > > > > > + /* Yield the floor on command completion */ > > > > > > + OUT_PKT7(ring, CP_CONTEXT_SWITCH_YIELD, 4); > > > > > > + OUT_RING(ring, 0x00); > > > > > > + OUT_RING(ring, 0x00); > > > > > > + OUT_RING(ring, 0x01); > > > > > > + /* Generate interrupt on preemption completion */ > > > > > > + OUT_RING(ring, 0x00); > > > > > > + > > > > > > + a6xx_flush(gpu, ring); > > > > > > + > > > > > > + return a6xx_idle(gpu, ring) ? 0 : -EINVAL; > > > > > > +} > > > > > > + > > > > > > static int a6xx_cp_init(struct msm_gpu *gpu) > > > > > > { > > > > > > struct msm_ringbuffer *ring = gpu->rb[0]; > > > > > > @@ -619,6 +893,8 @@ static int a6xx_cp_init(struct msm_gpu *gpu) > > > > > > > > > > > > static int a7xx_cp_init(struct msm_gpu *gpu) > > > > > > { > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > struct msm_ringbuffer *ring = gpu->rb[0]; > > > > > > u32 mask; > > > > > > > > > > > > @@ -626,6 +902,8 @@ static int a7xx_cp_init(struct msm_gpu *gpu) > > > > > > OUT_PKT7(ring, CP_THREAD_CONTROL, 1); > > > > > > OUT_RING(ring, BIT(27)); > > > > > > > > > > > > + a7xx_patch_pwrup_reglist(gpu); > > > > > > + > > > > > > OUT_PKT7(ring, CP_ME_INIT, 7); > > > > > > > > > > > > /* Use multiple HW contexts */ > > > > > > @@ -656,11 +934,11 @@ static int a7xx_cp_init(struct msm_gpu *gpu) > > > > > > > > > > > > /* *Don't* send a power up reg list for concurrent binning (TODO) */ > > > > > > /* Lo address */ > > > > > > - OUT_RING(ring, 0x00000000); > > > > > > + OUT_RING(ring, lower_32_bits(a6xx_gpu->pwrup_reglist_iova)); > > > > > > /* Hi address */ > > > > > > - OUT_RING(ring, 0x00000000); > > > > > > + OUT_RING(ring, upper_32_bits(a6xx_gpu->pwrup_reglist_iova)); > > > > > > /* BIT(31) set => read the regs from the list */ > > > > > > - OUT_RING(ring, 0x00000000); > > > > > > + OUT_RING(ring, BIT(31)); > > > > > > > > > > > > a6xx_flush(gpu, ring); > > > > > > return a6xx_idle(gpu, ring) ? 0 : -EINVAL; > > > > > > @@ -784,6 +1062,16 @@ static int a6xx_ucode_load(struct msm_gpu *gpu) > > > > > > msm_gem_object_set_name(a6xx_gpu->shadow_bo, "shadow"); > > > > > > } > > > > > > > > > > > > + a6xx_gpu->pwrup_reglist_ptr = msm_gem_kernel_new(gpu->dev, PAGE_SIZE, > > > > > > + MSM_BO_WC | MSM_BO_MAP_PRIV, > > > > > > + gpu->aspace, &a6xx_gpu->pwrup_reglist_bo, > > > > > > + &a6xx_gpu->pwrup_reglist_iova); > > > > > > + > > > > > > + if (IS_ERR(a6xx_gpu->pwrup_reglist_ptr)) > > > > > > + return PTR_ERR(a6xx_gpu->pwrup_reglist_ptr); > > > > > > + > > > > > > + msm_gem_object_set_name(a6xx_gpu->pwrup_reglist_bo, "pwrup_reglist"); > > > > > > + > > > > > > return 0; > > > > > > } > > > > > > > > > > > > @@ -1127,6 +1415,8 @@ static int hw_init(struct msm_gpu *gpu) > > > > > > if (a6xx_gpu->shadow_bo) { > > > > > > gpu_write64(gpu, REG_A6XX_CP_RB_RPTR_ADDR, > > > > > > shadowptr(a6xx_gpu, gpu->rb[0])); > > > > > > + for (unsigned int i = 0; i < gpu->nr_rings; i++) > > > > > > + a6xx_gpu->shadow[i] = 0; > > > > > > } > > > > > > > > > > > > /* ..which means "always" on A7xx, also for BV shadow */ > > > > > > @@ -1135,6 +1425,8 @@ static int hw_init(struct msm_gpu *gpu) > > > > > > rbmemptr(gpu->rb[0], bv_rptr)); > > > > > > } > > > > > > > > > > > > + a6xx_preempt_hw_init(gpu); > > > > > > + > > > > > > /* Always come up on rb 0 */ > > > > > > a6xx_gpu->cur_ring = gpu->rb[0]; > > > > > > > > > > > > @@ -1180,6 +1472,10 @@ static int hw_init(struct msm_gpu *gpu) > > > > > > out: > > > > > > if (adreno_has_gmu_wrapper(adreno_gpu)) > > > > > > return ret; > > > > > > + > > > > > > + /* Last step - yield the ringbuffer */ > > > > > > + a7xx_preempt_start(gpu); > > > > > > + > > > > > > /* > > > > > > * Tell the GMU that we are done touching the GPU and it can start power > > > > > > * management > > > > > > @@ -1557,8 +1853,13 @@ static irqreturn_t a6xx_irq(struct msm_gpu *gpu) > > > > > > if (status & A6XX_RBBM_INT_0_MASK_SWFUSEVIOLATION) > > > > > > a7xx_sw_fuse_violation_irq(gpu); > > > > > > > > > > > > - if (status & A6XX_RBBM_INT_0_MASK_CP_CACHE_FLUSH_TS) > > > > > > + if (status & A6XX_RBBM_INT_0_MASK_CP_CACHE_FLUSH_TS) { > > > > > > msm_gpu_retire(gpu); > > > > > > + a6xx_preempt_trigger(gpu); > > > > > > + } > > > > > > + > > > > > > + if (status & A6XX_RBBM_INT_0_MASK_CP_SW) > > > > > > + a6xx_preempt_irq(gpu); > > > > > > > > > > > > return IRQ_HANDLED; > > > > > > } > > > > > > @@ -2165,6 +2466,8 @@ static const struct adreno_gpu_funcs funcs = { > > > > > > .active_ring = a6xx_active_ring, > > > > > > .irq = a6xx_irq, > > > > > > .destroy = a6xx_destroy, > > > > > > + .submitqueue_setup = a6xx_preempt_submitqueue_setup, > > > > > > + .submitqueue_close = a6xx_preempt_submitqueue_close, > > > > > > #if defined(CONFIG_DRM_MSM_GPU_STATE) > > > > > > .show = a6xx_show, > > > > > > #endif > > > > > > @@ -2331,6 +2634,8 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev) > > > > > > a6xx_fault_handler); > > > > > > > > > > > > a6xx_calc_ubwc_config(adreno_gpu); > > > > > > + /* Set up the preemption specific bits and pieces for each ringbuffer */ > > > > > > + a6xx_preempt_init(gpu); > > > > > > > > > > > > return gpu; > > > > > > } > > > > > > diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu.h b/drivers/gpu/drm/msm/adreno/a6xx_gpu.h > > > > > > index e3e5c53ae8af..652e49f01428 100644 > > > > > > --- a/drivers/gpu/drm/msm/adreno/a6xx_gpu.h > > > > > > +++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu.h > > > > > > @@ -12,6 +12,31 @@ > > > > > > > > > > > > extern bool hang_debug; > > > > > > > > > > > > +struct cpu_gpu_lock { > > > > > > + uint32_t gpu_req; > > > > > > + uint32_t cpu_req; > > > > > > + uint32_t turn; > > > > > > + union { > > > > > > + struct { > > > > > > + uint16_t list_length; > > > > > > + uint16_t list_offset; > > > > > > + }; > > > > > > + struct { > > > > > > + uint8_t ifpc_list_len; > > > > > > + uint8_t preemption_list_len; > > > > > > + uint16_t dynamic_list_len; > > > > > > + }; > > > > > > + }; > > > > > > + uint64_t regs[62]; > > > > > > +}; > > > > > > + > > > > > > +struct adreno_reglist_list { > > > > > > + /** @reg: List of register **/ > > > > > > + const u32 *regs; > > > > > > + /** @count: Number of registers in the list **/ > > > > > > + u32 count; > > > > > > +}; > > > > > > + > > > > > > /** > > > > > > * struct a6xx_info - a6xx specific information from device table > > > > > > * > > > > > > @@ -31,6 +56,24 @@ struct a6xx_gpu { > > > > > > uint64_t sqe_iova; > > > > > > > > > > > > struct msm_ringbuffer *cur_ring; > > > > > > + struct msm_ringbuffer *next_ring; > > > > > > + > > > > > > + struct drm_gem_object *preempt_bo[MSM_GPU_MAX_RINGS]; > > > > > > + void *preempt[MSM_GPU_MAX_RINGS]; > > > > > > + uint64_t preempt_iova[MSM_GPU_MAX_RINGS]; > > > > > > + uint32_t last_seqno[MSM_GPU_MAX_RINGS]; > > > > > > + > > > > > > + atomic_t preempt_state; > > > > > > + spinlock_t eval_lock; > > > > > > + struct timer_list preempt_timer; > > > > > > + > > > > > > + unsigned int preempt_level; > > > > > > + bool uses_gmem; > > > > > > + bool skip_save_restore; > > > > > > + > > > > > > + struct drm_gem_object *scratch_bo; > > > > > > + void *scratch_ptr; > > > > > > + uint64_t scratch_iova; > > > > > > > > > > > > struct a6xx_gmu gmu; > > > > > > > > > > > > @@ -38,6 +81,10 @@ struct a6xx_gpu { > > > > > > uint64_t shadow_iova; > > > > > > uint32_t *shadow; > > > > > > > > > > > > + struct drm_gem_object *pwrup_reglist_bo; > > > > > > + void *pwrup_reglist_ptr; > > > > > > + uint64_t pwrup_reglist_iova; > > > > > > + > > > > > > bool has_whereami; > > > > > > > > > > > > void __iomem *llc_mmio; > > > > > > @@ -49,6 +96,108 @@ struct a6xx_gpu { > > > > > > > > > > > > #define to_a6xx_gpu(x) container_of(x, struct a6xx_gpu, base) > > > > > > > > > > > > +#define SCRATCH_USER_CTX_IOVA(ring_id, a6xx_gpu) \ > > > > > > + (a6xx_gpu->scratch_iova + (ring_id * sizeof(uint64_t))) > > > > > > + > > > > > > +/* > > > > > > + * In order to do lockless preemption we use a simple state machine to progress > > > > > > + * through the process. > > > > > > + * > > > > > > + * PREEMPT_NONE - no preemption in progress. Next state START. > > > > > > + * PREEMPT_START - The trigger is evaluating if preemption is possible. Next > > > > > > + * states: TRIGGERED, NONE > > > > > > + * PREEMPT_FINISH - An intermediate state before moving back to NONE. Next > > > > > > + * state: NONE. > > > > > > + * PREEMPT_TRIGGERED: A preemption has been executed on the hardware. Next > > > > > > + * states: FAULTED, PENDING > > > > > > + * PREEMPT_FAULTED: A preemption timed out (never completed). This will trigger > > > > > > + * recovery. Next state: N/A > > > > > > + * PREEMPT_PENDING: Preemption complete interrupt fired - the callback is > > > > > > + * checking the success of the operation. Next state: FAULTED, NONE. > > > > > > + */ > > > > > > + > > > > > > +enum a6xx_preempt_state { > > > > > > + PREEMPT_NONE = 0, > > > > > > + PREEMPT_START, > > > > > > + PREEMPT_FINISH, > > > > > > + PREEMPT_TRIGGERED, > > > > > > + PREEMPT_FAULTED, > > > > > > + PREEMPT_PENDING, > > > > > > +}; > > > > > > + > > > > > > +/* > > > > > > + * struct a6xx_preempt_record is a shared buffer between the microcode and the > > > > > > + * CPU to store the state for preemption. The record itself is much larger > > > > > > + * (2112k) but most of that is used by the CP for storage. > > > > > > + * > > > > > > + * There is a preemption record assigned per ringbuffer. When the CPU triggers a > > > > > > + * preemption, it fills out the record with the useful information (wptr, ring > > > > > > + * base, etc) and the microcode uses that information to set up the CP following > > > > > > + * the preemption. When a ring is switched out, the CP will save the ringbuffer > > > > > > + * state back to the record. In this way, once the records are properly set up > > > > > > + * the CPU can quickly switch back and forth between ringbuffers by only > > > > > > + * updating a few registers (often only the wptr). > > > > > > + * > > > > > > + * These are the CPU aware registers in the record: > > > > > > + * @magic: Must always be 0xAE399D6EUL > > > > > > + * @info: Type of the record - written 0 by the CPU, updated by the CP > > > > > > + * @errno: preemption error record > > > > > > + * @data: Data field in YIELD and SET_MARKER packets, Written and used by CP > > > > > > + * @cntl: Value of RB_CNTL written by CPU, save/restored by CP > > > > > > + * @rptr: Value of RB_RPTR written by CPU, save/restored by CP > > > > > > + * @wptr: Value of RB_WPTR written by CPU, save/restored by CP > > > > > > + * @_pad: Reserved/padding > > > > > > + * @rptr_addr: Value of RB_RPTR_ADDR_LO|HI written by CPU, save/restored by CP > > > > > > + * @rbase: Value of RB_BASE written by CPU, save/restored by CP > > > > > > + * @counter: GPU address of the storage area for the preemption counters > > > > > > + */ > > > > > > +struct a6xx_preempt_record { > > > > > > + u32 magic; > > > > > > + u32 info; > > > > > > + u32 errno; > > > > > > + u32 data; > > > > > > + u32 cntl; > > > > > > + u32 rptr; > > > > > > + u32 wptr; > > > > > > + u32 _pad; > > > > > > + u64 rptr_addr; > > > > > > + u64 rbase; > > > > > > + u64 counter; > > > > > > + u64 bv_rptr_addr; > > > > > > +}; > > > > > > + > > > > > > +#define A6XX_PREEMPT_RECORD_MAGIC 0xAE399D6EUL > > > > > > + > > > > > > +#define PREEMPT_RECORD_SIZE_FALLBACK(size) \ > > > > > > + ((size) == 0 ? 4192 * SZ_1K : (size)) > > > > > > + > > > > > > +#define PREEMPT_OFFSET_SMMU_INFO 0 > > > > > > +#define PREEMPT_OFFSET_PRIV_NON_SECURE (PREEMPT_OFFSET_SMMU_INFO + 4096) > > > > > > +#define PREEMPT_OFFSET_PRIV_SECURE(size) \ > > > > > > + (PREEMPT_OFFSET_PRIV_NON_SECURE + PREEMPT_RECORD_SIZE_FALLBACK(size)) > > > > > > +#define PREEMPT_SIZE(size) \ > > > > > > + (PREEMPT_OFFSET_PRIV_SECURE(size) + PREEMPT_RECORD_SIZE_FALLBACK(size)) > > > > > > + > > > > > > +/* > > > > > > + * The preemption counter block is a storage area for the value of the > > > > > > + * preemption counters that are saved immediately before context switch. We > > > > > > + * append it on to the end of the allocation for the preemption record. > > > > > > + */ > > > > > > +#define A6XX_PREEMPT_COUNTER_SIZE (16 * 4) > > > > > > + > > > > > > +#define A6XX_PREEMPT_USER_RECORD_SIZE (192 * 1024) > > > > > > + > > > > > > +struct a7xx_cp_smmu_info { > > > > > > + u32 magic; > > > > > > + u32 _pad4; > > > > > > + u64 ttbr0; > > > > > > + u32 asid; > > > > > > + u32 context_idr; > > > > > > + u32 context_bank; > > > > > > +}; > > > > > > + > > > > > > +#define GEN7_CP_SMMU_INFO_MAGIC 0x241350d5UL > > > > > > + > > > > > > /* > > > > > > * Given a register and a count, return a value to program into > > > > > > * REG_CP_PROTECT_REG(n) - this will block both reads and writes for > > > > > > @@ -106,6 +255,25 @@ int a6xx_gmu_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node); > > > > > > int a6xx_gmu_wrapper_init(struct a6xx_gpu *a6xx_gpu, struct device_node *node); > > > > > > void a6xx_gmu_remove(struct a6xx_gpu *a6xx_gpu); > > > > > > > > > > > > +void a6xx_preempt_init(struct msm_gpu *gpu); > > > > > > +void a6xx_preempt_hw_init(struct msm_gpu *gpu); > > > > > > +void a6xx_preempt_trigger(struct msm_gpu *gpu); > > > > > > +void a6xx_preempt_irq(struct msm_gpu *gpu); > > > > > > +void a6xx_preempt_fini(struct msm_gpu *gpu); > > > > > > +int a6xx_preempt_submitqueue_setup(struct msm_gpu *gpu, > > > > > > + struct msm_gpu_submitqueue *queue); > > > > > > +void a6xx_preempt_submitqueue_close(struct msm_gpu *gpu, > > > > > > + struct msm_gpu_submitqueue *queue); > > > > > > + > > > > > > +/* Return true if we are in a preempt state */ > > > > > > +static inline bool a6xx_in_preempt(struct a6xx_gpu *a6xx_gpu) > > > > > > +{ > > > > > > + int preempt_state = atomic_read(&a6xx_gpu->preempt_state); > > > > > > + > > > > > > + return !(preempt_state == PREEMPT_NONE || > > > > > > + preempt_state == PREEMPT_FINISH); > > > > > > +} > > > > > > + > > > > > > void a6xx_gmu_set_freq(struct msm_gpu *gpu, struct dev_pm_opp *opp, > > > > > > bool suspended); > > > > > > unsigned long a6xx_gmu_get_freq(struct msm_gpu *gpu); > > > > > > diff --git a/drivers/gpu/drm/msm/adreno/a6xx_preempt.c b/drivers/gpu/drm/msm/adreno/a6xx_preempt.c > > > > > > new file mode 100644 > > > > > > index 000000000000..0d402a3bcf5a > > > > > > --- /dev/null > > > > > > +++ b/drivers/gpu/drm/msm/adreno/a6xx_preempt.c > > > > > > @@ -0,0 +1,434 @@ > > > > > > +// SPDX-License-Identifier: GPL-2.0 > > > > > > +/* Copyright (c) 2018, The Linux Foundation. All rights reserved. */ > > > > > > +/* Copyright (c) 2023 Collabora, Ltd. */ > > > > > > +/* Copyright (c) 2024 Valve Corporation */ > > > > > > + > > > > > > +#include "msm_gem.h" > > > > > > +#include "a6xx_gpu.h" > > > > > > +#include "a6xx_gmu.xml.h" > > > > > > +#include "msm_mmu.h" > > > > > > + > > > > > > +#define FENCE_STATUS_WRITEDROPPED0_MASK 0x1 > > > > > > +#define FENCE_STATUS_WRITEDROPPED1_MASK 0x2 > > > > > > + > > > > > > +/* > > > > > > + * Try to transition the preemption state from old to new. Return > > > > > > + * true on success or false if the original state wasn't 'old' > > > > > > + */ > > > > > > +static inline bool try_preempt_state(struct a6xx_gpu *a6xx_gpu, > > > > > > + enum a6xx_preempt_state old, enum a6xx_preempt_state new) > > > > > > +{ > > > > > > + enum a6xx_preempt_state cur = atomic_cmpxchg(&a6xx_gpu->preempt_state, > > > > > > + old, new); > > > > > > + > > > > > > + return (cur == old); > > > > > > +} > > > > > > + > > > > > > +/* > > > > > > + * Force the preemption state to the specified state. This is used in cases > > > > > > + * where the current state is known and won't change > > > > > > + */ > > > > > > +static inline void set_preempt_state(struct a6xx_gpu *gpu, > > > > > > + enum a6xx_preempt_state new) > > > > > > +{ > > > > > > + /* > > > > > > + * preempt_state may be read by other cores trying to trigger a > > > > > > + * preemption or in the interrupt handler so barriers are needed > > > > > > + * before... > > > > > > + */ > > > > > > + smp_mb__before_atomic(); > > > > > > + atomic_set(&gpu->preempt_state, new); > > > > > > + /* ... and after*/ > > > > > > + smp_mb__after_atomic(); > > > > > > +} > > > > > > + > > > > > > +/* Write the most recent wptr for the given ring into the hardware */ > > > > > > +static inline void update_wptr(struct msm_gpu *gpu, struct msm_ringbuffer *ring) > > > > > > +{ > > > > > > + unsigned long flags; > > > > > > + uint32_t wptr, cur_wptr; > > > > > > + > > > > > > + if (!ring) > > > > > > + return; > > > > > > + > > > > > > + spin_lock_irqsave(&ring->preempt_lock, flags); > > > > > > + > > > > > > + if (ring->skip_inline_wptr) { > > > > > > + wptr = get_wptr(ring); > > > > > > + > > > > > > + gpu_write(gpu, REG_A6XX_CP_RB_WPTR, wptr); > > > > > > + > > > > > > + ring->skip_inline_wptr = false; > > > > > > + } > > > > > > + > > > > > > + spin_unlock_irqrestore(&ring->preempt_lock, flags); > > > > > > +} > > > > > > + > > > > > > +/* Return the highest priority ringbuffer with something in it */ > > > > > > +static struct msm_ringbuffer *get_next_ring(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + > > > > > > + unsigned long flags; > > > > > > + int i; > > > > > > + > > > > > > + for (i = 0; i < gpu->nr_rings; i++) { > > > > > > + bool empty; > > > > > > + struct msm_ringbuffer *ring = gpu->rb[i]; > > > > > > + > > > > > > + spin_lock_irqsave(&ring->preempt_lock, flags); > > > > > > + empty = (get_wptr(ring) == gpu->funcs->get_rptr(gpu, ring)); > > > > > > + if (!empty && ring == a6xx_gpu->cur_ring) > > > > > > + empty = ring->memptrs->fence == a6xx_gpu->last_seqno[i]; > > > > > > + spin_unlock_irqrestore(&ring->preempt_lock, flags); > > > > > > + > > > > > > + if (!empty) > > > > > > + return ring; > > > > > > + } > > > > > > + > > > > > > + return NULL; > > > > > > +} > > > > > > + > > > > > > +static void a6xx_preempt_timer(struct timer_list *t) > > > > > > +{ > > > > > > + struct a6xx_gpu *a6xx_gpu = from_timer(a6xx_gpu, t, preempt_timer); > > > > > > + struct msm_gpu *gpu = &a6xx_gpu->base.base; > > > > > > + struct drm_device *dev = gpu->dev; > > > > > > + > > > > > > + if (!try_preempt_state(a6xx_gpu, PREEMPT_TRIGGERED, PREEMPT_FAULTED)) > > > > > > + return; > > > > > > + > > > > > > + dev_err(dev->dev, "%s: preemption timed out\n", gpu->name); > > > > > > + kthread_queue_work(gpu->worker, &gpu->recover_work); > > > > > > +} > > > > > > + > > > > > > +void a6xx_preempt_irq(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + uint32_t status; > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + struct drm_device *dev = gpu->dev; > > > > > > + > > > > > > + if (!try_preempt_state(a6xx_gpu, PREEMPT_TRIGGERED, PREEMPT_PENDING)) > > > > > > + return; > > > > > > + > > > > > > + /* Delete the preemption watchdog timer */ > > > > > > + del_timer(&a6xx_gpu->preempt_timer); > > > > > > + > > > > > > + /* > > > > > > + * The hardware should be setting the stop bit of CP_CONTEXT_SWITCH_CNTL > > > > > > + * to zero before firing the interrupt, but there is a non zero chance > > > > > > + * of a hardware condition or a software race that could set it again > > > > > > + * before we have a chance to finish. If that happens, log and go for > > > > > > + * recovery > > > > > > + */ > > > > > > + status = gpu_read(gpu, REG_A6XX_CP_CONTEXT_SWITCH_CNTL); > > > > > > + if (unlikely(status & 0x1)) { > > > > > > + DRM_DEV_ERROR(&gpu->pdev->dev, > > > > > > + "!!!!!!!!!!!!!!!! preemption faulted !!!!!!!!!!!!!! irq\n"); > > > > > > + set_preempt_state(a6xx_gpu, PREEMPT_FAULTED); > > > > > > + dev_err(dev->dev, "%s: Preemption failed to complete\n", > > > > > > + gpu->name); > > > > > > + kthread_queue_work(gpu->worker, &gpu->recover_work); > > > > > > + return; > > > > > > + } > > > > > > + > > > > > > + a6xx_gpu->cur_ring = a6xx_gpu->next_ring; > > > > > > + a6xx_gpu->next_ring = NULL; > > > > > > + > > > > > > + /* Make sure the write to cur_ring is posted before the change in state */ > > > > > > + wmb(); > > > > > > + > > > > > > + set_preempt_state(a6xx_gpu, PREEMPT_FINISH); > > > > > > + > > > > > > + update_wptr(gpu, a6xx_gpu->cur_ring); > > > > > > + > > > > > > + set_preempt_state(a6xx_gpu, PREEMPT_NONE); > > > > > > + > > > > > > + /* > > > > > > + * Retrigger preemption to avoid a deadlock that might occur when preemption > > > > > > + * is skipped due to it being already in flight when requested. > > > > > > + */ > > > > > > + a6xx_preempt_trigger(gpu); > > > > > > +} > > > > > > + > > > > > > +void a6xx_preempt_hw_init(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + int i; > > > > > > + > > > > > > + /* No preemption if we only have one ring */ > > > > > > + if (gpu->nr_rings == 1) > > > > > > + return; > > > > > > + > > > > > > + for (i = 0; i < gpu->nr_rings; i++) { > > > > > > + struct a6xx_preempt_record *record_ptr = > > > > > > + a6xx_gpu->preempt[i] + PREEMPT_OFFSET_PRIV_NON_SECURE; > > > > > > + record_ptr->wptr = 0; > > > > > > + record_ptr->rptr = 0; > > > > > > + record_ptr->rptr_addr = shadowptr(a6xx_gpu, gpu->rb[i]); > > > > > > + record_ptr->info = 0; > > > > > > + record_ptr->data = 0; > > > > > > + record_ptr->rbase = gpu->rb[i]->iova; > > > > > > + } > > > > > > + > > > > > > + /* Write a 0 to signal that we aren't switching pagetables */ > > > > > > + gpu_write64(gpu, REG_A6XX_CP_CONTEXT_SWITCH_SMMU_INFO, 0); > > > > > > + > > > > > > + /* Enable the GMEM save/restore feature for preemption */ > > > > > > + gpu_write(gpu, REG_A6XX_RB_CONTEXT_SWITCH_GMEM_SAVE_RESTORE, 0x1); > > > > > > + > > > > > > + /* Reset the preemption state */ > > > > > > + set_preempt_state(a6xx_gpu, PREEMPT_NONE); > > > > > > + > > > > > > + spin_lock_init(&a6xx_gpu->eval_lock); > > > > > > + > > > > > > + /* Always come up on rb 0 */ > > > > > > + a6xx_gpu->cur_ring = gpu->rb[0]; > > > > > > +} > > > > > > + > > > > > > +void a6xx_preempt_trigger(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + u64 preempt_offset_priv_secure; > > > > > > + unsigned long flags; > > > > > > + struct msm_ringbuffer *ring; > > > > > > + uint64_t user_ctx_iova; > > > > > > + unsigned int cntl; > > > > > > + > > > > > > + if (gpu->nr_rings == 1) > > > > > > + return; > > > > > > + > > > > > > + /* > > > > > > + * Lock to make sure another thread attempting preemption doesn't skip it > > > > > > + * while we are still evaluating the next ring. This makes sure the other > > > > > > + * thread does start preemption if we abort it and avoids a soft lock. > > > > > > + */ > > > > > > + spin_lock_irqsave(&a6xx_gpu->eval_lock, flags); > > > > > > + > > > > > > + /* > > > > > > + * Try to start preemption by moving from NONE to START. If > > > > > > + * unsuccessful, a preemption is already in flight > > > > > > + */ > > > > > > + if (!try_preempt_state(a6xx_gpu, PREEMPT_NONE, PREEMPT_START)) { > > > > > > + spin_unlock_irqrestore(&a6xx_gpu->eval_lock, flags); > > > > > > + return; > > > > > > + } > > > > > > + > > > > > > + cntl = (((a6xx_gpu->preempt_level << 6) & 0xC0) | > > > > > > + ((a6xx_gpu->skip_save_restore << 9) & 0x200) | > > > > > > + ((a6xx_gpu->uses_gmem << 8) & 0x100) | 0x1); > > > > > > + > > > > > > + /* Get the next ring to preempt to */ > > > > > > + ring = get_next_ring(gpu); > > > > > > + > > > > > > + /* > > > > > > + * If no ring is populated or the highest priority ring is the current > > > > > > + * one do nothing except to update the wptr to the latest and greatest > > > > > > + */ > > > > > > + if (!ring || (a6xx_gpu->cur_ring == ring)) { > > > > > > + set_preempt_state(a6xx_gpu, PREEMPT_FINISH); > > > > > > + update_wptr(gpu, a6xx_gpu->cur_ring); > > > > > > + set_preempt_state(a6xx_gpu, PREEMPT_NONE); > > > > > > + spin_unlock_irqrestore(&a6xx_gpu->eval_lock, flags); > > > > > > + return; > > > > > > + } > > > > > > + > > > > > > + spin_unlock_irqrestore(&a6xx_gpu->eval_lock, flags); > > > > > > + > > > > > > + spin_lock_irqsave(&ring->preempt_lock, flags); > > > > > > + > > > > > > + struct a7xx_cp_smmu_info *smmu_info_ptr = > > > > > > + a6xx_gpu->preempt[ring->id] + PREEMPT_OFFSET_SMMU_INFO; > > > > > > + struct a6xx_preempt_record *record_ptr = > > > > > > + a6xx_gpu->preempt[ring->id] + PREEMPT_OFFSET_PRIV_NON_SECURE; > > > > > > + u64 ttbr0 = ring->memptrs->ttbr0; > > > > > > + u32 context_idr = ring->memptrs->context_idr; > > > > > > + > > > > > > + smmu_info_ptr->ttbr0 = ttbr0; > > > > > > + smmu_info_ptr->context_idr = context_idr; > > > > > > + record_ptr->wptr = get_wptr(ring); > > > > > > + > > > > > > + /* > > > > > > + * The GPU will write the wptr we set above when we preempt. Reset > > > > > > + * skip_inline_wptr to make sure that we don't write WPTR to the same > > > > > > + * thing twice. It's still possible subsequent submissions will update > > > > > > + * wptr again, in which case they will set the flag to true. This has > > > > > > + * to be protected by the lock for setting the flag and updating wptr > > > > > > + * to be atomic. > > > > > > + */ > > > > > > + ring->skip_inline_wptr = false; > > > > > > + > > > > > > + spin_unlock_irqrestore(&ring->preempt_lock, flags); > > > > > > + > > > > > > + gpu_write64(gpu, > > > > > > + REG_A6XX_CP_CONTEXT_SWITCH_SMMU_INFO, > > > > > > + a6xx_gpu->preempt_iova[ring->id] + PREEMPT_OFFSET_SMMU_INFO); > > > > > > + > > > > > > + gpu_write64(gpu, > > > > > > + REG_A6XX_CP_CONTEXT_SWITCH_PRIV_NON_SECURE_RESTORE_ADDR, > > > > > > + a6xx_gpu->preempt_iova[ring->id] + PREEMPT_OFFSET_PRIV_NON_SECURE); > > > > > > + > > > > > > + preempt_offset_priv_secure = > > > > > > + PREEMPT_OFFSET_PRIV_SECURE(adreno_gpu->info->preempt_record_size); > > > > > > + gpu_write64(gpu, > > > > > > + REG_A6XX_CP_CONTEXT_SWITCH_PRIV_SECURE_RESTORE_ADDR, > > > > > > + a6xx_gpu->preempt_iova[ring->id] + preempt_offset_priv_secure); > > > > > > + > > > > > > + /* > > > > > > + * Use the user context iova from the scratch memory that the CP may > > > > > > + * have written as part of the ring switch out. > > > > > > + */ > > > > > > + user_ctx_iova = *((uint64_t *)a6xx_gpu->scratch_ptr + ring->id); > > > > > > + > > > > > > + gpu_write64(gpu, > > > > > > + REG_A6XX_CP_CONTEXT_SWITCH_NON_PRIV_RESTORE_ADDR, > > > > > > + user_ctx_iova); > > > > > > + > > > > > > + a6xx_gpu->next_ring = ring; > > > > > > + > > > > > > + /* Start a timer to catch a stuck preemption */ > > > > > > + mod_timer(&a6xx_gpu->preempt_timer, jiffies + msecs_to_jiffies(10000)); > > > > > > + > > > > > > + /* Set the preemption state to triggered */ > > > > > > + set_preempt_state(a6xx_gpu, PREEMPT_TRIGGERED); > > > > > > + > > > > > > + /* Make sure any previous writes to WPTR are posted */ > > > > > > + gpu_read(gpu, REG_A6XX_CP_RB_WPTR); > > > > > > + > > > > > > + /* Make sure everything is written before hitting the button */ > > > > > > + wmb(); > > > > > > + > > > > > > + /* Trigger the preemption */ > > > > > > + gpu_write(gpu, REG_A6XX_CP_CONTEXT_SWITCH_CNTL, cntl); > > > > > > +} > > > > > > + > > > > > > +static int preempt_init_ring(struct a6xx_gpu *a6xx_gpu, > > > > > > + struct msm_ringbuffer *ring) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = &a6xx_gpu->base; > > > > > > + struct msm_gpu *gpu = &adreno_gpu->base; > > > > > > + struct drm_gem_object *bo = NULL; > > > > > > + phys_addr_t ttbr; > > > > > > + u64 iova = 0; > > > > > > + void *ptr; > > > > > > + int asid; > > > > > > + > > > > > > + ptr = msm_gem_kernel_new(gpu->dev, > > > > > > + PREEMPT_SIZE(adreno_gpu->info->preempt_record_size), > > > > > > + MSM_BO_WC | MSM_BO_MAP_PRIV, gpu->aspace, &bo, &iova); > > > > > > + > > > > > > + memset(ptr, 0, PREEMPT_SIZE(adreno_gpu->info->preempt_record_size)); > > > > > > + > > > > > > + if (IS_ERR(ptr)) > > > > > > + return PTR_ERR(ptr); > > > > > > + > > > > > > + a6xx_gpu->preempt_bo[ring->id] = bo; > > > > > > + a6xx_gpu->preempt_iova[ring->id] = iova; > > > > > > + a6xx_gpu->preempt[ring->id] = ptr; > > > > > > + > > > > > > + struct a7xx_cp_smmu_info *smmu_info_ptr = ptr + PREEMPT_OFFSET_SMMU_INFO; > > > > > > + struct a6xx_preempt_record *record_ptr = ptr + PREEMPT_OFFSET_PRIV_NON_SECURE; > > > > > > + > > > > > > + msm_iommu_pagetable_params(gpu->aspace->mmu, &ttbr, &asid); > > > > > > + > > > > > > + smmu_info_ptr->magic = GEN7_CP_SMMU_INFO_MAGIC; > > > > > > + smmu_info_ptr->ttbr0 = ttbr; > > > > > > + smmu_info_ptr->asid = 0xdecafbad; > > > > > > + smmu_info_ptr->context_idr = 0; > > > > > > + > > > > > > + /* Set up the defaults on the preemption record */ > > > > > > + record_ptr->magic = A6XX_PREEMPT_RECORD_MAGIC; > > > > > > + record_ptr->info = 0; > > > > > > + record_ptr->data = 0; > > > > > > + record_ptr->rptr = 0; > > > > > > + record_ptr->wptr = 0; > > > > > > + record_ptr->cntl = MSM_GPU_RB_CNTL_DEFAULT; > > > > > > + record_ptr->rbase = ring->iova; > > > > > > + record_ptr->counter = 0; > > > > > > + record_ptr->bv_rptr_addr = rbmemptr(ring, bv_rptr); > > > > > > + > > > > > > + return 0; > > > > > > +} > > > > > > + > > > > > > +void a6xx_preempt_fini(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + int i; > > > > > > + > > > > > > + for (i = 0; i < gpu->nr_rings; i++) > > > > > > + msm_gem_kernel_put(a6xx_gpu->preempt_bo[i], gpu->aspace); > > > > > > +} > > > > > > + > > > > > > +void a6xx_preempt_init(struct msm_gpu *gpu) > > > > > > +{ > > > > > > + struct adreno_gpu *adreno_gpu = to_adreno_gpu(gpu); > > > > > > + struct a6xx_gpu *a6xx_gpu = to_a6xx_gpu(adreno_gpu); > > > > > > + int i; > > > > > > + > > > > > > + /* No preemption if we only have one ring */ > > > > > > + if (gpu->nr_rings <= 1) > > > > > > + return; > > > > > > + > > > > > > + for (i = 0; i < gpu->nr_rings; i++) { > > > > > > + if (preempt_init_ring(a6xx_gpu, gpu->rb[i])) > > > > > > + goto fail; > > > > > > + } > > > > > > + > > > > > > + /* TODO: make this configurable? */ > > > > > > + a6xx_gpu->preempt_level = 1; > > > > > > + a6xx_gpu->uses_gmem = 1; > > > > > > + a6xx_gpu->skip_save_restore = 1; > > > > > > + > > > > > > + a6xx_gpu->scratch_ptr = msm_gem_kernel_new(gpu->dev, > > > > > > + gpu->nr_rings * sizeof(uint64_t), MSM_BO_WC, > > > > > > + gpu->aspace, &a6xx_gpu->scratch_bo, > > > > > > + &a6xx_gpu->scratch_iova); > > > > > > + > > > > > > + if (IS_ERR(a6xx_gpu->scratch_ptr)) > > > > > > + goto fail; > > > > > > + > > > > > > + timer_setup(&a6xx_gpu->preempt_timer, a6xx_preempt_timer, 0); > > > > > > + > > > > > > + return; > > > > > > +fail: > > > > > > + /* > > > > > > + * On any failure our adventure is over. Clean up and > > > > > > + * set nr_rings to 1 to force preemption off > > > > > > + */ > > > > > > + a6xx_preempt_fini(gpu); > > > > > > + gpu->nr_rings = 1; > > > > > > + > > > > > > + return; > > > > > > +} > > > > > > + > > > > > > +void a6xx_preempt_submitqueue_close(struct msm_gpu *gpu, > > > > > > + struct msm_gpu_submitqueue *queue) > > > > > > +{ > > > > > > + if (!queue->bo) > > > > > > + return; > > > > > > + > > > > > > + msm_gem_kernel_put(queue->bo, gpu->aspace); > > > > > > +} > > > > > > + > > > > > > +int a6xx_preempt_submitqueue_setup(struct msm_gpu *gpu, > > > > > > + struct msm_gpu_submitqueue *queue) > > > > > > +{ > > > > > > + void *ptr; > > > > > > + > > > > > > + /* > > > > > > + * Create a per submitqueue buffer for the CP to save and restore user > > > > > > + * specific information such as the VPC streamout data. > > > > > > + */ > > > > > > + ptr = msm_gem_kernel_new(gpu->dev, A6XX_PREEMPT_USER_RECORD_SIZE, > > > > > > + MSM_BO_WC, gpu->aspace, &queue->bo, &queue->bo_iova); > > > > > > + > > > > > > + if (IS_ERR(ptr)) > > > > > > + return PTR_ERR(ptr); > > > > > > + > > > > > > + return 0; > > > > > > +} > > > > > > diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.h b/drivers/gpu/drm/msm/msm_ringbuffer.h > > > > > > index 40791b2ade46..7dde6a312511 100644 > > > > > > --- a/drivers/gpu/drm/msm/msm_ringbuffer.h > > > > > > +++ b/drivers/gpu/drm/msm/msm_ringbuffer.h > > > > > > @@ -36,6 +36,7 @@ struct msm_rbmemptrs { > > > > > > > > > > > > volatile struct msm_gpu_submit_stats stats[MSM_GPU_SUBMIT_STATS_COUNT]; > > > > > > volatile u64 ttbr0; > > > > > > + volatile u32 context_idr; > > > > > > }; > > > > > > > > > > > > struct msm_cp_state { > > > > > > @@ -100,6 +101,12 @@ struct msm_ringbuffer { > > > > > > * preemption. Can be aquired from irq context. > > > > > > */ > > > > > > spinlock_t preempt_lock; > > > > > > + > > > > > > + /* > > > > > > + * Whether we skipped writing wptr and it needs to be updated in the > > > > > > + * future when the ring becomes current. > > > > > > + */ > > > > > > + bool skip_inline_wptr; > > > > > > }; > > > > > > > > > > > > struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, > > > > > > > > > > > > -- > > > > > > 2.46.0 > > > > > > > > > > > >