On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@xxxxxxx>
wrote:
Alex wrote:
On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@xxxxxxx>
wrote:
Hi Alex,
We have not validated virtualization of our integrated
GPUs. I
don't
know that it will work at all. We had done a bit of
testing but
ran
into the same issues with the PSP, but never had a chance
to
debug
further because this feature is not productized.
...
You need a functional PSP to get the GPU driver up and
running.
Ah, thanks for the hint :)
I guess that if I want to have any chance to get the PSP
working
I'm
going to need more details on it. A quick search some time
ago
mostly
brought reverse-engineering work, rather than official AMD
doc.
Are
there some AMD resources I missed ?
The driver code is pretty much it.
Let's try to shed some more light on how things work, taking as
excuse
psp_v12_0_ring_create().
First, register access through [RW]REG32_SOC15() is implemented
in
terms of __[RW]REG32_SOC15_RLC__(), which is basically a
[RW]REG32(),
except it has to be more complex in the SR-IOV case.
Has the RLC anything to do with SR-IOV ?
When running the driver on a SR-IOV virtual function (VF), some
registers are not available directly via the VF's MMIO aperture so
they need to go through the RLC. For bare metal or passthrough
this
is not relevant.
It accesses registers in the MMIO range of the MP0 IP, and the
"MP0"
name correlates highly with MMIO accesses in PSP-handling code.
Is "MP0" another name for PSP (and "MP1" for SMU) ? The MP0
version
Yes.
reported at v11.0.3 by discovery seems to contradict the use of
v12.0
for RENOIR as set by soc15_set_ip_blocks(), or do I miss
something ?
Typo in the ip discovery table on renoir.
More generally (and mostly out of curiosity while we're at it),
do we
have a way to match IPs listed at discovery time with the ones
used
in the driver ?
In general, barring typos, the code is shared at the major version
level. The actual code may or may not need changes to handle minor
revision changes in an IP. The driver maps the IP versions from
the
ip discovery table to the code contained in the driver.
---
As for the register names, maybe we could have a short
explanation of
how they are structured ? Eg. mmMP0_SMN_C2PMSG_69: that seems to
be
a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not
sure
of the "SMN" part -- that could refer to the "System Management
Network",
described in [0] as an internal bus. Are we accessing this
register
through this SMN ?
These registers are just mailboxes for the PSP firmware. All of
the
C2PMSG registers functionality is defined by the PSP firmware.
They
are basically scratch registers used to communicate between the
driver
and the PSP firmware.
On APUs, the PSP is shared with
the CPU and the rest of the platform. The GPU driver just
interacts
with it for a few specific tasks:
1. Loading Trusted Applications (e.g., trusted firmware
applications
that run on the PSP for specific functionality, e.g., HDCP and
content
protection, etc.)
2. Validating and loading firmware for other engines on the
SoC.
This
is required to use those engines.
Trying to understand in more details how we start the PSP up, I
noticed
that psp_v12_0 has support for loading a sOS firmware, but never
calls
init_sos_microcode() - and anyway there is no sos firmware for
renoir
and green_sardine, which seem to be the only ASICs with this PSP
version.
Is it something that's just not been completely wired up yet ?
On APUs, the PSP is shared with the CPU so the PSP firmware is part
of
the sbios image. The driver doesn't load it. We only load it on
dGPUs where the driver is responsible for the chip initialization.
That also rings a bell, that we have nothing about Secure OS in
the doc
yet (not even the acronym in the glossary).
I'm not too familiar with the PSP's path to memory from the GPU
perspective. IIRC, most memory used by the PSP goes through
carve
out
"vram" on APUs so it should work, but I would double check if
there
are any system memory allocations that used to interact with
the PSP
and see if changing them to vram helps. It does work with the
IOMMU
enabled on bare metal, so it should work in passthrough as well
in
theory.
I can see a single case in the PSP code where GTT is used instead
of
vram: to create fw_pri_bo when SR-IOV is not used (and there has
to be a reason, since the SR-IOV code path does use vram).
Changing it to vram does not make a difference, but then the
only bo that seems to be used at that point is the one for the
psp ring,
which is allocated in vram, so I'm not too much surprised.
Maybe I should double-check bo_create calls to hunt for more ?
We looked into this a bit ourselves and ran into the same issues.
We'd probably need to debug this with the PSP team to make further
progress, but this was not productized so neither team had the
resources to delve further.
Alex
[0]
https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf
