On Sun, Dec 19, 2021 at 11:41 AM Yann Dirson <ydirson@xxxxxxx> wrote: > > Christian wrote: > > Am 19.12.21 um 17:00 schrieb Yann Dirson: > > > Alex wrote: > > >> Thinking about this more, I think the problem might be related to > > >> CPU > > >> access to "VRAM". APUs don't have dedicated VRAM, they use a > > >> reserved > > >> carve out region at the top of system memory. For CPU access to > > >> this > > >> memory, we kmap the physical address of the carve out region of > > >> system > > >> memory. You'll need to make sure that region is accessible to the > > >> guest. > > > So basically, the non-virt flow is is: (video?) BIOS reserves > > > memory, marks it > > > as reserved in e820, stores the physaddr somewhere, which the GPU > > > driver gets. > > > Since I suppose this includes the framebuffer, this probably has to > > > occur around > > > the moment the driver calls > > > drm_aperture_remove_conflicting_pci_framebuffers() > > > (which happens before this hw init step), right ? > > > > Well, that partially correct. The efifb is using the PCIe resources > > to > > access the framebuffer and as far as I know we use that one to kick > > it out. > > > > The stolen memory we get over e820/registers is separate to that. > > > > > ... which brings me to a point that's been puzzling me for some > > > time, which is > > > that as the hw init fails, the efifb driver is still using the > > > framebuffer. > > > > No, it isn't. You are probably just still seeing the same screen. > > > > The issue is most likely that while efi was kicked out nobody > > re-programmed the display hardware to show something different. > > > > > Am I right in suspecting that efifb should get stripped of its > > > ownership of the > > > fb aperture first, and that if I don't get a black screen on > > > hw_init failure > > > that issue should be the first focus point ? > > > > You assumption with the black screen is incorrect. Since the hardware > > works independent even if you kick out efi you still have the same > > screen content, you just can't update it anymore. > > It's not only that the screen keeps its contents, it's that the dom0 > happily continues updating it. If the hypevisor is using efifb, then yes that could be a problem as the hypervisor could be writing to the efifb resources which ends up writing to the same physical memory. That applies to any GPU on a UEFI system. You'll need to make sure efifb is not in use in the hypervisor. Alex > > > But putting efi asside what Alex pointed out pretty much breaks your > > neck trying to forward the device. You maybe could try to hack the > > driver to use the PCIe BAR for framebuffer access, but that might be > > quite a bit slower. > > > > Regards, > > Christian. > > > > > > > >> Alex > > >> > > >> On Mon, Dec 13, 2021 at 3:29 PM Alex Deucher > > >> <alexdeucher@xxxxxxxxx> > > >> wrote: > > >>> On Sun, Dec 12, 2021 at 5:19 PM Yann Dirson <ydirson@xxxxxxx> > > >>> wrote: > > >>>> Alex wrote: > > >>>>> On Mon, Dec 6, 2021 at 4:36 PM Yann Dirson <ydirson@xxxxxxx> > > >>>>> wrote: > > >>>>>> Hi Alex, > > >>>>>> > > >>>>>>> We have not validated virtualization of our integrated > > >>>>>>> GPUs. I > > >>>>>>> don't > > >>>>>>> know that it will work at all. We had done a bit of > > >>>>>>> testing but > > >>>>>>> ran > > >>>>>>> into the same issues with the PSP, but never had a chance > > >>>>>>> to > > >>>>>>> debug > > >>>>>>> further because this feature is not productized. > > >>>>>> ... > > >>>>>>> You need a functional PSP to get the GPU driver up and > > >>>>>>> running. > > >>>>>> Ah, thanks for the hint :) > > >>>>>> > > >>>>>> I guess that if I want to have any chance to get the PSP > > >>>>>> working > > >>>>>> I'm > > >>>>>> going to need more details on it. A quick search some time > > >>>>>> ago > > >>>>>> mostly > > >>>>>> brought reverse-engineering work, rather than official AMD > > >>>>>> doc. > > >>>>>> Are > > >>>>>> there some AMD resources I missed ? > > >>>>> The driver code is pretty much it. > > >>>> Let's try to shed some more light on how things work, taking as > > >>>> excuse > > >>>> psp_v12_0_ring_create(). > > >>>> > > >>>> First, register access through [RW]REG32_SOC15() is implemented > > >>>> in > > >>>> terms of __[RW]REG32_SOC15_RLC__(), which is basically a > > >>>> [RW]REG32(), > > >>>> except it has to be more complex in the SR-IOV case. > > >>>> Has the RLC anything to do with SR-IOV ? > > >>> When running the driver on a SR-IOV virtual function (VF), some > > >>> registers are not available directly via the VF's MMIO aperture > > >>> so > > >>> they need to go through the RLC. For bare metal or passthrough > > >>> this > > >>> is not relevant. > > >>> > > >>>> It accesses registers in the MMIO range of the MP0 IP, and the > > >>>> "MP0" > > >>>> name correlates highly with MMIO accesses in PSP-handling code. > > >>>> Is "MP0" another name for PSP (and "MP1" for SMU) ? The MP0 > > >>>> version > > >>> Yes. > > >>> > > >>>> reported at v11.0.3 by discovery seems to contradict the use of > > >>>> v12.0 > > >>>> for RENOIR as set by soc15_set_ip_blocks(), or do I miss > > >>>> something ? > > >>> Typo in the ip discovery table on renoir. > > >>> > > >>>> More generally (and mostly out of curiosity while we're at it), > > >>>> do we > > >>>> have a way to match IPs listed at discovery time with the ones > > >>>> used > > >>>> in the driver ? > > >>> In general, barring typos, the code is shared at the major > > >>> version > > >>> level. The actual code may or may not need changes to handle > > >>> minor > > >>> revision changes in an IP. The driver maps the IP versions from > > >>> the > > >>> ip discovery table to the code contained in the driver. > > >>> > > >>>> --- > > >>>> > > >>>> As for the register names, maybe we could have a short > > >>>> explanation of > > >>>> how they are structured ? Eg. mmMP0_SMN_C2PMSG_69: that seems > > >>>> to > > >>>> be > > >>>> a MMIO register named "C2PMSG_69" in the "MP0" IP, but I'm not > > >>>> sure > > >>>> of the "SMN" part -- that could refer to the "System Management > > >>>> Network", > > >>>> described in [0] as an internal bus. Are we accessing this > > >>>> register > > >>>> through this SMN ? > > >>> These registers are just mailboxes for the PSP firmware. All of > > >>> the > > >>> C2PMSG registers functionality is defined by the PSP firmware. > > >>> They > > >>> are basically scratch registers used to communicate between the > > >>> driver > > >>> and the PSP firmware. > > >>> > > >>>> > > >>>>> On APUs, the PSP is shared with > > >>>>> the CPU and the rest of the platform. The GPU driver just > > >>>>> interacts > > >>>>> with it for a few specific tasks: > > >>>>> 1. Loading Trusted Applications (e.g., trusted firmware > > >>>>> applications > > >>>>> that run on the PSP for specific functionality, e.g., HDCP and > > >>>>> content > > >>>>> protection, etc.) > > >>>>> 2. Validating and loading firmware for other engines on the > > >>>>> SoC. > > >>>>> This > > >>>>> is required to use those engines. > > >>>> Trying to understand in more details how we start the PSP up, I > > >>>> noticed > > >>>> that psp_v12_0 has support for loading a sOS firmware, but never > > >>>> calls > > >>>> init_sos_microcode() - and anyway there is no sos firmware for > > >>>> renoir > > >>>> and green_sardine, which seem to be the only ASICs with this PSP > > >>>> version. > > >>>> Is it something that's just not been completely wired up yet ? > > >>> On APUs, the PSP is shared with the CPU so the PSP firmware is > > >>> part > > >>> of > > >>> the sbios image. The driver doesn't load it. We only load it on > > >>> dGPUs where the driver is responsible for the chip > > >>> initialization. > > >>> > > >>>> That also rings a bell, that we have nothing about Secure OS in > > >>>> the doc > > >>>> yet (not even the acronym in the glossary). > > >>>> > > >>>> > > >>>>> I'm not too familiar with the PSP's path to memory from the GPU > > >>>>> perspective. IIRC, most memory used by the PSP goes through > > >>>>> carve > > >>>>> out > > >>>>> "vram" on APUs so it should work, but I would double check if > > >>>>> there > > >>>>> are any system memory allocations that used to interact with > > >>>>> the PSP > > >>>>> and see if changing them to vram helps. It does work with the > > >>>>> IOMMU > > >>>>> enabled on bare metal, so it should work in passthrough as well > > >>>>> in > > >>>>> theory. > > >>>> I can see a single case in the PSP code where GTT is used > > >>>> instead > > >>>> of > > >>>> vram: to create fw_pri_bo when SR-IOV is not used (and there has > > >>>> to be a reason, since the SR-IOV code path does use vram). > > >>>> Changing it to vram does not make a difference, but then the > > >>>> only bo that seems to be used at that point is the one for the > > >>>> psp ring, > > >>>> which is allocated in vram, so I'm not too much surprised. > > >>>> > > >>>> Maybe I should double-check bo_create calls to hunt for more ? > > >>> We looked into this a bit ourselves and ran into the same issues. > > >>> We'd probably need to debug this with the PSP team to make > > >>> further > > >>> progress, but this was not productized so neither team had the > > >>> resources to delve further. > > >>> > > >>> Alex > > >>> > > >>>> > > >>>> [0] > > >>>> https://github.com/PSPReverse/psp-docs/blob/master/masterthesis-eichner-psp-2020.pdf > > > >
