Re: [PATCH 3/4] drm/amdgpu: Add kernel parameter for ignoring bad page threshold

[Christian König <ckoenig.leichtzumerken@xxxxxxxxx>]

> As it stands, we have at least two customers who are focused on having the threshold automatically remove the GPUs from use, to ensure data integrity. They just want warnings to know that it's getting bad (my 90% threshold patch), so that they can plan for HW replacement accordingly.

We could handle that outside of the kernel driver. E.g. in userspace for 
example.

Not loading the driver at all results in numerous problems and I think 
not able to retrieve or reset the bad pages counter is just the tip of 
the iceberg here.

To be honest I'm pretty sure that rejecting the driver to load if it 
would work is a no-go and that design is really questionable.

Christian.

Am 20.10.21 um 16:56 schrieb Russell, Kent:
> [AMD Official Use Only]
>
> I can see both sides of the argument. Having a configurable threshold means that you can determine what sort of "HW reliability" that you want. The default value is likely not going to get hit by the average user. And users that DO hit that threshold can determine if they want to ignore it, increase it, or replace the hardware, through the kernel parameter.
>
> But having that option means it's configurable based on what that customer wants. If they believe that all data is precious, setting the threshold to something like 1 bad page means that they won't ever run on a chip that ever had a bad page, thus ensuring data integrity. It seems ludicrous to me to have a value so low, but I am sure that someone out there would want to remove a GPU as soon as it has one bad page retired. And some people couldn't care any less, so they can set it to disabled or ignored or whatever they wish.
>
> As it stands, we have at least two customers who are focused on having the threshold automatically remove the GPUs from use, to ensure data integrity. They just want warnings to know that it's getting bad (my 90% threshold patch), so that they can plan for HW replacement accordingly.
>
>   Kent
>
> > -----Original Message-----
> > From: Christian König <ckoenig.leichtzumerken@xxxxxxxxx>
> > Sent: Wednesday, October 20, 2021 6:55 AM
> > To: Russell, Kent <Kent.Russell@xxxxxxx>; amd-gfx@xxxxxxxxxxxxxxxxxxxxx
> > Cc: Tuikov, Luben <Luben.Tuikov@xxxxxxx>; Joshi, Mukul <Mukul.Joshi@xxxxxxx>
> > Subject: Re: [PATCH 3/4] drm/amdgpu: Add kernel parameter for ignoring bad page
> > threshold
> >
> > Am 19.10.21 um 19:50 schrieb Kent Russell:
> > > When a GPU hits the bad_page_threshold, it will not be initialized by
> > > the amdgpu driver. This means that the table cannot be cleared, nor can
> > > information gathering be performed (getting serial number, BDF, etc).
> > > Add an override called ignore_bad_page_threshold that can be set to true
> > > to still initialize the GPU, even when the bad page threshold has been
> > > reached.
> > I would rather question the practice of this bad pages threshold.
> >
> > As far as I know the hardware works perfectly fine even when we have
> > more bad badles then expected, we should just warn really loudly about it.
> >
> > Christian.
> >
> > > Cc: Luben Tuikov <luben.tuikov@xxxxxxx>
> > > Cc: Mukul Joshi <Mukul.Joshi@xxxxxxx>
> > > Signed-off-by: Kent Russell <kent.russell@xxxxxxx>
> > > ---
> > >    drivers/gpu/drm/amd/amdgpu/amdgpu.h     |  1 +
> > >    drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 13 +++++++++++++
> > >    2 files changed, 14 insertions(+)
> > >
> > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > > index d58e37fd01f4..b85b67a88a3d 100644
> > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
> > > @@ -205,6 +205,7 @@ extern struct amdgpu_mgpu_info mgpu_info;
> > >    extern int amdgpu_ras_enable;
> > >    extern uint amdgpu_ras_mask;
> > >    extern int amdgpu_bad_page_threshold;
> > > +extern bool amdgpu_ignore_bad_page_threshold;
> > >    extern struct amdgpu_watchdog_timer amdgpu_watchdog_timer;
> > >    extern int amdgpu_async_gfx_ring;
> > >    extern int amdgpu_mcbp;
> > > diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > index 96bd63aeeddd..3e9a7b072888 100644
> > > --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
> > > @@ -189,6 +189,7 @@ struct amdgpu_mgpu_info mgpu_info = {
> > >    int amdgpu_ras_enable = -1;
> > >    uint amdgpu_ras_mask = 0xffffffff;
> > >    int amdgpu_bad_page_threshold = -1;
> > > +bool amdgpu_ignore_bad_page_threshold;
> > >    struct amdgpu_watchdog_timer amdgpu_watchdog_timer = {
> > >    	.timeout_fatal_disable = false,
> > >    	.period = 0x0, /* default to 0x0 (timeout disable) */
> > > @@ -880,6 +881,18 @@ module_param_named(reset_method, amdgpu_reset_method,
> > int, 0444);
> > >    MODULE_PARM_DESC(bad_page_threshold, "Bad page threshold(-1 = auto(default
> > value), 0 = disable bad page retirement)");
> > >    module_param_named(bad_page_threshold, amdgpu_bad_page_threshold, int, 0444);
> > >
> > > +/**
> > > + * DOC: ignore_bad_page_threshold (bool) Bad page threshold specifies
> > > + * the threshold value of faulty pages detected by RAS ECC. Once the
> > > + * threshold is hit, the GPU will not be initialized. Use this parameter
> > > + * to ignore the bad page threshold so that information gathering can
> > > + * still be performed. This also allows for booting the GPU to clear
> > > + * the RAS EEPROM table.
> > > + */
> > > +
> > > +MODULE_PARM_DESC(ignore_bad_page_threshold, "Ignore bad page threshold (false =
> > respect bad page threshold (default value)");
> > > +module_param_named(ignore_bad_page_threshold,
> > amdgpu_ignore_bad_page_threshold, bool, 0644);
> > > +
> > >    MODULE_PARM_DESC(num_kcq, "number of kernel compute queue user want to setup
> > (8 if set to greater than 8 or less than 0, only affect gfx 8+)");
> > >    module_param_named(num_kcq, amdgpu_num_kcq, int, 0444);