On Mon, Sep 13, 2021 at 10:09:56PM -0700, Matthew Brost wrote:
> From: John Harrison <John.C.Harrison@xxxxxxxxx>
>
> When i915 receives a context reset notification from GuC, it triggers
> an error capture before resetting any outstanding requsts of that
> context. Unfortunately, the error capture is not a time bound
> operation. In certain situations it can take a long time, particularly
> when multiple large LMEM buffers must be read back and eoncoded. If
> this delay is longer than other timeouts (heartbeat, test recovery,
> etc.) then a full GT reset can be triggered in the middle.
>
> That can result in the context being reset by GuC actually being
> destroyed before the error capture completes and the GuC submission
> code resumes. Thus, the GuC side can start dereferencing stale
> pointers and Bad Things ensue.
>
> So add a refcount get of the context during the entire reset
> operation. That way, the context can't be destroyed part way through
> no matter what other resets or user interactions occur.
>
> v2:
> (Matthew Brost)
> - Update patch to work with async error capture
>
> Signed-off-by: John Harrison <John.C.Harrison@xxxxxxxxx>
> Signed-off-by: Matthew Brost <matthew.brost@xxxxxxxxx>
This sounds like a fundamental issue in our reset/scheduler design. If we
have multiple timeout-things working in parallel, then there's going to be
an endless whack-a-mole fireworks show.
Reset is not a perf critical path (aside from media timeout, which guc
handles internally anyway). Simplicity trumps everything else. The fix
here is to guarantee that anything related to reset cannot happen in
parallel with anything else related to reset/timeout. At least on a
per-engine (and really on a per-reset domain) basis.
The fix we've developed for drm/sched is that the driver can allocate a
single-thread work queue, pass it to each drm/sched instance, and all
timeout handling is run in there.
For i915 it's more of a mess since we have a ton of random things that
time out/reset potentially going on in parallel. But that's the design we
should head towards.
_not_ sprinkling random refcounts all over the place until most of the
oops/splats disappear. That's cargo-culting, not engineering.
-Daniel
> ---
> .../gpu/drm/i915/gt/uc/intel_guc_submission.c | 24 +++++++++++++++++--
> 1 file changed, 22 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c
> index 1986a57b52cc..02917fc4d4a8 100644
> --- a/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c
> +++ b/drivers/gpu/drm/i915/gt/uc/intel_guc_submission.c
> @@ -2888,6 +2888,8 @@ static void capture_worker_func(struct work_struct *w)
> intel_engine_set_hung_context(engine, ce);
> with_intel_runtime_pm(&i915->runtime_pm, wakeref)
> i915_capture_error_state(gt, ce->engine->mask);
> +
> + intel_context_put(ce);
> }
>
> static void capture_error_state(struct intel_guc *guc,
> @@ -2924,7 +2926,7 @@ static void guc_context_replay(struct intel_context *ce)
> tasklet_hi_schedule(&sched_engine->tasklet);
> }
>
> -static void guc_handle_context_reset(struct intel_guc *guc,
> +static bool guc_handle_context_reset(struct intel_guc *guc,
> struct intel_context *ce)
> {
> trace_intel_context_reset(ce);
> @@ -2937,7 +2939,11 @@ static void guc_handle_context_reset(struct intel_guc *guc,
> !context_blocked(ce))) {
> capture_error_state(guc, ce);
> guc_context_replay(ce);
> +
> + return false;
> }
> +
> + return true;
> }
>
> int intel_guc_context_reset_process_msg(struct intel_guc *guc,
> @@ -2945,6 +2951,7 @@ int intel_guc_context_reset_process_msg(struct intel_guc *guc,
> {
> struct intel_context *ce;
> int desc_idx;
> + unsigned long flags;
>
> if (unlikely(len != 1)) {
> drm_err(&guc_to_gt(guc)->i915->drm, "Invalid length %u", len);
> @@ -2952,11 +2959,24 @@ int intel_guc_context_reset_process_msg(struct intel_guc *guc,
> }
>
> desc_idx = msg[0];
> +
> + /*
> + * The context lookup uses the xarray but lookups only require an RCU lock
> + * not the full spinlock. So take the lock explicitly and keep it until the
> + * context has been reference count locked to ensure it can't be destroyed
> + * asynchronously until the reset is done.
> + */
> + xa_lock_irqsave(&guc->context_lookup, flags);
> ce = g2h_context_lookup(guc, desc_idx);
> + if (ce)
> + intel_context_get(ce);
> + xa_unlock_irqrestore(&guc->context_lookup, flags);
> +
> if (unlikely(!ce))
> return -EPROTO;
>
> - guc_handle_context_reset(guc, ce);
> + if (guc_handle_context_reset(guc, ce))
> + intel_context_put(ce);
>
> return 0;
> }
> --
> 2.32.0
>
--
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
