On Tue, Nov 22, 2022 at 6:53 AM Robert Beckett <bob.beckett@xxxxxxxxxxxxx> wrote: > > Hi, > > > does anyone know any documentation, or can provide advice on debugging amdgpu fault reports? This is a GPU page fault so it refers the the GPU virtual address space of the application . Each process (well fd really), gets its own GPU virtual address space into which system memory, system mmio space, or vram can be mapped. The user mode drivers control their GPU virtual address space. > > > e.g: > > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:8 vmid:1 pasid:32769, for process vkcube pid 999 thread vkcube pid 999) This is the process that caused the fault. > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: in page starting at address 0x0000800100700000 from client 0x1b (UTCL2) This is the virtual address that faulted. > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00101A10 > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: Faulty UTCL2 client ID: SDMA0 (0xd) The fault came from the SDMA engine. > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: MORE_FAULTS: 0x0 > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: WALKER_ERROR: 0x0 > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: PERMISSION_FAULTS: 0x1 The page was not marked as valid in the GPU page table. > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: MAPPING_ERROR: 0x0 > Nov 21 19:17:06 fedora kernel: amdgpu 0000:01:00.0: amdgpu: RW: 0x0 SDMA attempted to read an invalid page. > > > > see https://gitlab.freedesktop.org/drm/amd/-/issues/2267 for more context. > > We have a complicated setup involving rendering then blitting to virtio-gpu exported dmabufs, with plenty of hacks in the mesa and xwayland stacks, so we are considering this our issue to debug, and not an issue with the driver at this point. > However, having debugged all the interesting parts leading to these faults, I am unable to decode the fault report to correlate to a buffer. > > in the fault report, what address space is the address from? > given that the fault handler shifts the reported addres up by 12, I assume it is a 4K pfn which makes me assume a physical address is this correct? > if so, is that a vram pa or a host system memory pa? > is there any documentation for the other fields reported like the fault status etc? See the comments above. There is some kernel doc as well: https://docs.kernel.org/gpu/amdgpu/driver-core.html#amdgpu-virtual-memory > > I'd appreciate any advice you could give to help us debug further. Some operation you are doing in the user mode driver is reading an invalid page. Possibly reading past the end of a buffer or something mis-aligned. Compare the faulting GPU address to the GPU virtual address space in the application and you should be able to track down what is happening. Alex > > Thanks > > Bob >
