On 9/10/2021 11:04 PM, Caleb Connolly wrote:
On 10/09/2021 18:18, Rob Clark wrote:
On Tue, Sep 7, 2021 at 7:20 PM Bjorn Andersson
<bjorn.andersson@xxxxxxxxxx> wrote:
On Mon 09 Aug 10:26 PDT 2021, Akhil P Oommen wrote:
On 8/9/2021 9:48 PM, Caleb Connolly wrote:
On 09/08/2021 17:12, Rob Clark wrote:
On Mon, Aug 9, 2021 at 7:52 AM Akhil P Oommen
<akhilpo@xxxxxxxxxxxxxx> wrote:
[..]
I am a bit confused. We don't define a power domain for gpu in dt,
correct? Then what exactly set_opp do here? Do you think this
usleep is
what is helping here somehow to mask the issue?
The power domains (for cx and gx) are defined in the GMU DT, the
OPPs in
the GPU DT. For the sake of simplicity I'll refer to the lowest
frequency (257000000) and OPP level (RPMH_REGULATOR_LEVEL_LOW_SVS) as
the "min" state, and the highest frequency (710000000) and OPP level
(RPMH_REGULATOR_LEVEL_TURBO_L1) as the "max" state. These are
defined in
sdm845.dtsi under the gpu node.
The new devfreq behaviour unmasks what I think is a driver bug, it
inadvertently puts much more strain on the GPU regulators than they
usually get. With the new behaviour the GPU jumps from it's min
state to
the max state and back again extremely rapidly under workloads as
small
as refreshing UI. Where previously the GPU would rarely if ever go
above
342MHz when interacting with the device, it now jumps between min and
max many times per second.
If my understanding is correct, the current implementation of the GMU
set freq is the following:
- Get OPP for frequency to set
- Push the frequency to the GMU - immediately updating the core
clock
- Call dev_pm_opp_set_opp() which triggers a notify chain, this
winds
up somewhere in power management code and causes the gx regulator
level
to be updated
Nope. dev_pm_opp_set_opp() sets the bandwidth for gpu and nothing
else. We
were using a different api earlier which got deprecated -
dev_pm_opp_set_bw().
On the Lenovo Yoga C630 this is reproduced by starting alacritty and if
I'm lucky I managed to hit a few keys before it crashes, so I spent a
few hours looking into this as well...
As you say, the dev_pm_opp_set_opp() will only cast a interconnect vote.
The opp-level is just there for show and isn't used by anything, at
least not on 845.
Further more, I'm missing something in my tree, so the interconnect
doesn't hit sync_state, and as such we're not actually scaling the
buses. So the problem is not that Linux doesn't turn on the buses in
time.
So I suspect that the "AHB bus error" isn't saying that we turned off
the bus, but rather that the GPU becomes unstable or something of that
sort.
Lastly, I reverted 9bc95570175a ("drm/msm: Devfreq tuning") and ran
Aquarium for 20 minutes without a problem. I then switched the gpu
devfreq governor to "userspace" and ran the following:
while true; do
echo 257000000 > /sys/class/devfreq/5000000.gpu/userspace/set_freq
echo 710000000 > /sys/class/devfreq/5000000.gpu/userspace/set_freq
done
It took 19 iterations of this loop to crash the GPU.
I assume you still had aquarium running, to keep the gpu awake while
you ran that loop?
Fwiw, I modified this slightly to match sc7180's min/max gpu freq and
could not trigger any issue.. interestingly sc7180 has a lower min
freq (180) and higher max freq (800) so it was toggling over a wider
freq range. I also tried on a device that had the higher 825MHz opp
(since I noticed that was the only opp that used
RPMH_REGULATOR_LEVEL_TURBO_L1 and wanted to rule that out), but could
not reproduce.
I guess a630 (sdm845) should have higher power draw (it is 2x # of
shader cores and 2x GMEM size, but lower max freq).. the question is,
is this the reason we see this on sdm845 and not sc7180? Or is there
some other difference. On the gpu side of this, they are both closely
related (ie. the same "sub-generation" of a6xx, same gmu fw, etc)..
I'm less sure about the other parts (icc, rpmh, etc)
My guess would be power draw, nobody has mentioned this yet but I've
realised that the vdd_gfx rail is powered by a buck converter, which
could explain a lot of the symptoms.
Buck converters depend on high frequency switching and inductors to
work, this inherently leads to some lag time when changing voltages, and
also means that the behaviour of the regulator is defined in part by how
much current is being drawn. Wikipedia has a pretty good explanation:
https://en.wikipedia.org/wiki/Buck_converter
At the best of times these regulators have a known voltage ripple, when
under load and when rapidly switching voltages this will get a lot worse.
Someone with an oscilloscope and schematics could probe the rail and
probably see exactly what's going on when the GPU crashes. Because of
the lag time in the regulator changing voltage, it might be
undershooting whilst the GPU is trying to clock up and draw more current
- causing instability and crashes.
Both of you are correct. The GPU is very similar including the GMU (we
have same fw for both), except the GBIF block. As far as I am aware, the
non-gpu blocks within SoC should be similar except the configs.
And yes, for these sort of issues where we suspect a power issue, gx
rail should be probed for droops using a very high resolution
oscilloscopes (these droops might last less than 1us).
I am aware of only Dragonboard that is still alive from QC perspective.
Can someone report this issue to DB support team as it is fairly easy to
reproduce?
-Akhil.
BR,
-R
So the problem doesn't seem to be Rob's change, it's just that prior to
it the chance to hitting it is way lower. Question is still what it is
that we're triggering.
Regards,
Bjorn
