Christian wrote: > Am 19.12.21 um 17:00 schrieb Yann Dirson: > > Alex wrote: > >> Thinking about this more, I think the problem might be related to > >> CPU > >> access to "VRAM". APUs don't have dedicated VRAM, they use a > >> reserved > >> carve out region at the top of system memory. For CPU access to > >> this > >> memory, we kmap the physical address of the carve out region of > >> system > >> memory. You'll need to make sure that region is accessible to the > >> guest. > > So basically, the non-virt flow is is: (video?) BIOS reserves > > memory, marks it > > as reserved in e820, stores the physaddr somewhere, which the GPU > > driver gets. > > Since I suppose this includes the framebuffer, this probably has to > > occur around > > the moment the driver calls > > drm_aperture_remove_conflicting_pci_framebuffers() > > (which happens before this hw init step), right ? > > Well, that partially correct. The efifb is using the PCIe resources > to > access the framebuffer and as far as I know we use that one to kick > it out. > > The stolen memory we get over e820/registers is separate to that. > > > ... which brings me to a point that's been puzzling me for some > > time, which is > > that as the hw init fails, the efifb driver is still using the > > framebuffer. > > No, it isn't. You are probably just still seeing the same screen. > > The issue is most likely that while efi was kicked out nobody > re-programmed the display hardware to show something different. > > > Am I right in suspecting that efifb should get stripped of its > > ownership of the > > fb aperture first, and that if I don't get a black screen on > > hw_init failure > > that issue should be the first focus point ? > > You assumption with the black screen is incorrect. Since the hardware > works independent even if you kick out efi you still have the same > screen content, you just can't update it anymore. It's not only that the screen keeps its contents, it's that the dom0 happily continues updating it. > But putting efi asside what Alex pointed out pretty much breaks your > neck trying to forward the device. You maybe could try to hack the > driver to use the PCIe BAR for framebuffer access, but that might be > quite a bit slower. > > Regards, > Christian. > > > > >> Alex > >> > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher > >> <alexdeucher@xxxxxxxxx> > >> wrote: > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@xxxxxxx> > >>> wrote: > >>>> Alex wrote: > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@xxxxxxx> > >>>>> wrote: > >>>>>> Hi Alex, > >>>>>> > >>>>>>> We have not validated virtualization of our integrated > >>>>>>> GPUs. I > >>>>>>> don't > >>>>>>> know that it will work at all. We had done a bit of > >>>>>>> testing but > >>>>>>> ran > >>>>>>> into the same issues with the PSP, but never had a chance > >>>>>>> to > >>>>>>> debug > >>>>>>> further because this feature is not productized. > >>>>>> ... > >>>>>>> You need a functional PSP to get the GPU driver up and > >>>>>>> running. > >>>>>> Ah, thanks for the hint :) > >>>>>> > >>>>>> I guess that if I want to have any chance to get the PSP > >>>>>> working > >>>>>> I'm > >>>>>> going to need more details on it. A quick search some time > >>>>>> ago > >>>>>> mostly > >>>>>> brought reverse-engineering work, rather than official AMD > >>>>>> doc. > >>>>>> Are > >>>>>> there some AMD resources I missed ? > >>>>> The driver code is pretty much it. > >>>> Let's try to shed some more light on how things work, taking as > >>>> excuse > >>>> psp_v12_0_ring_create(). > >>>> > >>>> First, register access through [RW]REG32_SOC15() is implemented > >>>> in > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a > >>>> [RW]REG32(), > >>>> except it has to be more complex in the SR-IOV case. > >>>> Has the RLC anything to do with SR-IOV ? > >>> When running the driver on a SR-IOV virtual function (VF), some > >>> registers are not available directly via the VF's MMIO aperture > >>> so > >>> they need to go through the RLC. For bare metal or passthrough > >>> this > >>> is not relevant. > >>> > >>>> It accesses registers in the MMIO range of the MP0 IP, and the > >>>> "MP0" > >>>> name correlates highly with MMIO accesses in PSP-handling code. > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ? The MP0 > >>>> version > >>> Yes. > >>> > >>>> reported at v11.0.3 by discovery seems to contradict the use of > >>>> v12.0 > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss > >>>> something ? > >>> Typo in the ip discovery table on renoir. > >>> > >>>> More generally (and mostly out of curiosity while we're at it), > >>>> do we > >>>> have a way to match IPs listed at discovery time with the ones > >>>> used > >>>> in the driver ? > >>> In general, barring typos, the code is shared at the major > >>> version > >>> level. The actual code may or may not need changes to handle > >>> minor > >>> revision changes in an IP. The driver maps the IP versions from > >>> the > >>> ip discovery table to the code contained in the driver. > >>> > >>>> --- > >>>> > >>>> As for the register names, maybe we could have a short > >>>> explanation of > >>>> how they are structured ? Eg. mmMP0_SMN_C2PMSG_69: that seems > >>>> to > >>>> be > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not > >>>> sure > >>>> of the "SMN" part -- that could refer to the "System Management > >>>> Network", > >>>> described in [0] as an internal bus. Are we accessing this > >>>> register > >>>> through this SMN ? > >>> These registers are just mailboxes for the PSP firmware. All of > >>> the > >>> C2PMSG registers functionality is defined by the PSP firmware. > >>> They > >>> are basically scratch registers used to communicate between the > >>> driver > >>> and the PSP firmware. > >>> > >>>> > >>>>> On APUs, the PSP is shared with > >>>>> the CPU and the rest of the platform. The GPU driver just > >>>>> interacts > >>>>> with it for a few specific tasks: > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware > >>>>> applications > >>>>> that run on the PSP for specific functionality, e.g., HDCP and > >>>>> content > >>>>> protection, etc.) > >>>>> 2. Validating and loading firmware for other engines on the > >>>>> SoC. > >>>>> This > >>>>> is required to use those engines. > >>>> Trying to understand in more details how we start the PSP up, I > >>>> noticed > >>>> that psp_v12_0 has support for loading a sOS firmware, but never > >>>> calls > >>>> init_sos_microcode() - and anyway there is no sos firmware for > >>>> renoir > >>>> and green_sardine, which seem to be the only ASICs with this PSP > >>>> version. > >>>> Is it something that's just not been completely wired up yet ? > >>> On APUs, the PSP is shared with the CPU so the PSP firmware is > >>> part > >>> of > >>> the sbios image. The driver doesn't load it. We only load it on > >>> dGPUs where the driver is responsible for the chip > >>> initialization. > >>> > >>>> That also rings a bell, that we have nothing about Secure OS in > >>>> the doc > >>>> yet (not even the acronym in the glossary). > >>>> > >>>> > >>>>> I'm not too familiar with the PSP's path to memory from the GPU > >>>>> perspective. IIRC, most memory used by the PSP goes through > >>>>> carve > >>>>> out > >>>>> "vram" on APUs so it should work, but I would double check if > >>>>> there > >>>>> are any system memory allocations that used to interact with > >>>>> the PSP > >>>>> and see if changing them to vram helps. It does work with the > >>>>> IOMMU > >>>>> enabled on bare metal, so it should work in passthrough as well > >>>>> in > >>>>> theory. > >>>> I can see a single case in the PSP code where GTT is used > >>>> instead > >>>> of > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and there has > >>>> to be a reason, since the SR-IOV code path does use vram). > >>>> Changing it to vram does not make a difference, but then the > >>>> only bo that seems to be used at that point is the one for the > >>>> psp ring, > >>>> which is allocated in vram, so I'm not too much surprised. > >>>> > >>>> Maybe I should double-check bo_create calls to hunt for more ? > >>> We looked into this a bit ourselves and ran into the same issues. > >>> We'd probably need to debug this with the PSP team to make > >>> further > >>> progress, but this was not productized so neither team had the > >>> resources to delve further. > >>> > >>> Alex > >>> > >>>> > >>>> [0] > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf > >
