On Mon, Sep 6, 2021 at 1:50 PM Rob Clark <robdclark@xxxxxxxxx> wrote: > > On Mon, Sep 6, 2021 at 12:58 PM Amit Pundir <amit.pundir@xxxxxxxxxx> wrote: > > > > On Mon, 6 Sept 2021 at 21:54, Rob Clark <robdclark@xxxxxxxxx> wrote: > > > > > > On Mon, Sep 6, 2021 at 1:02 AM Amit Pundir <amit.pundir@xxxxxxxxxx> wrote: > > > > > > > > On Sat, 4 Sept 2021 at 01:55, Rob Clark <robdclark@xxxxxxxxx> wrote: > > > > > > > > > > On Fri, Sep 3, 2021 at 12:39 PM John Stultz <john.stultz@xxxxxxxxxx> wrote: > > > > > > > > > > > > On Thu, Jul 29, 2021 at 1:49 PM Rob Clark <robdclark@xxxxxxxxx> wrote: > > > > > > > On Thu, Jul 29, 2021 at 1:28 PM Caleb Connolly > > > > > > > <caleb.connolly@xxxxxxxxxx> wrote: > > > > > > > > On 29/07/2021 21:24, Rob Clark wrote: > > > > > > > > > On Thu, Jul 29, 2021 at 1:06 PM Caleb Connolly > > > > > > > > > <caleb.connolly@xxxxxxxxxx> wrote: > > > > > > > > >> > > > > > > > > >> Hi Rob, > > > > > > > > >> > > > > > > > > >> I've done some more testing! It looks like before that patch ("drm/msm: Devfreq tuning") the GPU would never get above > > > > > > > > >> the second frequency in the OPP table (342MHz) (at least, not in glxgears). With the patch applied it would more > > > > > > > > >> aggressively jump up to the max frequency which seems to be unstable at the default regulator voltages. > > > > > > > > > > > > > > > > > > *ohh*, yeah, ok, that would explain it > > > > > > > > > > > > > > > > > >> Hacking the pm8005 s1 regulator (which provides VDD_GFX) up to 0.988v (instead of the stock 0.516v) makes the GPU stable > > > > > > > > >> at the higher frequencies. > > > > > > > > >> > > > > > > > > >> Applying this patch reverts the behaviour, and the GPU never goes above 342MHz in glxgears, losing ~30% performance in > > > > > > > > >> glxgear. > > > > > > > > >> > > > > > > > > >> I think (?) that enabling CPR support would be the proper solution to this - that would ensure that the regulators run > > > > > > > > >> at the voltage the hardware needs to be stable. > > > > > > > > >> > > > > > > > > >> Is hacking the voltage higher (although ideally not quite that high) an acceptable short term solution until we have > > > > > > > > >> CPR? Or would it be safer to just not make use of the higher frequencies on a630 for now? > > > > > > > > >> > > > > > > > > > > > > > > > > > > tbh, I'm not sure about the regulator stuff and CPR.. Bjorn is already > > > > > > > > > on CC and I added sboyd, maybe one of them knows better. > > > > > > > > > > > > > > > > > > In the short term, removing the higher problematic OPPs from dts might > > > > > > > > > be a better option than this patch (which I'm dropping), since there > > > > > > > > > is nothing stopping other workloads from hitting higher OPPs. > > > > > > > > Oh yeah that sounds like a more sensible workaround than mine . > > > > > > > > > > > > > > > > > > I'm slightly curious why I didn't have problems at higher OPPs on my > > > > > > > > > c630 laptop (sdm850) > > > > > > > > Perhaps you won the sillicon lottery - iirc sdm850 is binned for higher clocks as is out of the factory. > > > > > > > > > > > > > > > > Would it be best to drop the OPPs for all devices? Or just those affected? I guess it's possible another c630 might > > > > > > > > crash where yours doesn't? > > > > > > > > > > > > > > I've not heard any reports of similar issues from the handful of other > > > > > > > folks with c630's on #aarch64-laptops.. but I can't really say if that > > > > > > > is luck or not. > > > > > > > > > > > > > > Maybe just remove it for affected devices? But I'll defer to Bjorn. > > > > > > > > > > > > Just as another datapoint, I was just marveling at how suddenly smooth > > > > > > the UI was performing on db845c and Caleb pointed me at the "drm/msm: > > > > > > Devfreq tuning" patch as the likely cause of the improvement, and > > > > > > mid-discussion my board crashed into USB crash mode: > > > > > > [ 146.157696][ C0] adreno 5000000.gpu: CP | AHB bus error > > > > > > [ 146.163303][ C0] adreno 5000000.gpu: CP | AHB bus error > > > > > > [ 146.168837][ C0] adreno 5000000.gpu: RBBM | ATB bus overflow > > > > > > [ 146.174960][ C0] adreno 5000000.gpu: CP | HW fault | status=0x00000000 > > > > > > [ 146.181917][ C0] adreno 5000000.gpu: CP | AHB bus error > > > > > > [ 146.187547][ C0] adreno 5000000.gpu: CP illegal instruction error > > > > > > [ 146.194009][ C0] adreno 5000000.gpu: CP | AHB bus error > > > > > > [ 146.308909][ T9] Internal error: synchronous external abort: > > > > > > 96000010 [#1] PREEMPT SMP > > > > > > [ 146.317150][ T9] Modules linked in: > > > > > > [ 146.320941][ T9] CPU: 3 PID: 9 Comm: kworker/u16:1 Tainted: G > > > > > > W 5.14.0-mainline-06795-g42b258c2275c #24 > > > > > > [ 146.331974][ T9] Hardware name: Thundercomm Dragonboar > > > > > > Format: Log Type - Time(microsec) - Message - Optional Info > > > > > > Log Type: B - Since Boot(Power On Reset), D - Delta, S - Statistic > > > > > > S - QC_IMAGE_VERSION_STRING=BOOT.XF.2.0-00371-SDM845LZB-1 > > > > > > S - IMAGE_VARIANT_STRING=SDM845LA > > > > > > S - OEM_IMAGE_VERSION_STRING=TSBJ-FA-PC-02170 > > > > > > > > > > > > So Caleb sent me to this thread. :) > > > > > > > > > > > > I'm still trying to trip it again, but it does seem like db845c is > > > > > > also seeing some stability issues with Linus' HEAD. > > > > > > > > > > > > > > > > Caleb's original pastebin seems to have expired (or at least require > > > > > some sort of ubuntu login to access).. were the crashes he was seeing > > > > > also 'AHB bus error'? > > > > > > > > I can reproduce this hard crash > > > > https://www.irccloud.com/pastebin/Cu6UJntE/ and a gpu lockup > > > > https://www.irccloud.com/pastebin/6Ryd2Pug/ at times reliably, by > > > > running antutu benchmark on pocof1. > > > > > > > > Reverting 9bc95570175a ("drm/msm: Devfreq tuning") helps and I no > > > > longer see these errors. > > > > > > > > Complete dmesg for hardcrash https://pastebin.com/raw/GLZVQFQN > > > > > > > > > > Does antutu trigger this issue as easily on db845c? If no, does > > > db845c have pmic differences compared to pocof1 and Caleb's phone? > > > > Yes I can reproduce this hard crash with antutu on db845c as well with > > linux/master at 477f70cd2a67 ("Merge tag 'drm-next-2021-08-31-1' of > > git://anongit.freedesktop.org/drm/drm"). > > > > Dmesg: https://pastebin.com/raw/xXtvxk0G > > > > ok, I guess it is at least not a board specific thing (ie. won't need > to introduce some dt binding).. > > It would be nice to know what the maximum we can safely increase freq > in one step, if we need to limit that. Also, one sanity check.. for android builds, are you using the same a630_gmu.bin from linux-firmware? If not, does the l-f gmu fw change things? For freq changes, we basically ask gmu for the freq we want, and it votes for the requested freq.. so a gmu fw bug could be possible here. BR, -R > > > > > > > > > I think we may need some help from qcom here, but I'll go back and > > > look at older downstream kernels to see if I can find any evidence > > > that we need to limit how far we change the freq in a single step. > > > It's not clear to me if there is some physical constraint that the > > > driver needs to respect, or if we have some missing/incorrect > > > configuration for a630. IIRC the downstream kernel is letting the GMU > > > do more of the freq management, so it might be handling this case for > > > the kernel. But the GMU is a bit of a black box to me and I don't > > > have any docs, so just a guess. > > > > > > It would be helpful if someone who can repro this could try the > > > experiments I mentioned about increasing min_freq and/or decreasing > > > max_freq to limit the size of the freq change until the issue does not > > > happen. > > > > > > If we have to, we can merge this hack patch to disable freq clamping > > > on a630.. but that isn't really a fix. The root issue is a power > > > issue, 9bc95570175a just made it more likely to see the problem. > > > > > > BR, > > > -R > > > > > > > Regards, > > > > Amit Pundir > > > > > > > > > > > > > > If you have a reliable reproducer, I guess it would be worth seeing if > > > > > increasing the min_freq (ie. to limit how far we jump the freq in one > > > > > shot) "fixes" it? > > > > > > > > > > I guess I could check downstream kgsl to see if they were doing > > > > > something to increase freq in smaller increments.. I don't recall that > > > > > they were but it has been a while since I dug thru that code. And I > > > > > suppose downstream it could also be done in their custom tz governor. > > > > > > > > > > BR, > > > > > -R