On 24/10/2022 17:58, Christoph Hellwig wrote: > On Mon, Oct 24, 2022 at 03:20:43PM +0200, Juergen Gross wrote: >> Dom0 is (normally) a PV domain, so the physical memory can be still above >> 4 GB even with dom0_mem set to 4GB. > Which means that we need to ensure the DMA ops for Xen-PV (which is > always xen-swiotlb I think?) need to return DMA_BIT_MASK(64) or whatever > is the highest possible address. Xen tries to allocate memory from the top down, in order to keep the low memory free for special purses. (32bit PV guests strictly need to live below the 128G boundary in host physical space for ABI reasons, and Xen's struct vcpu needs to be below the 4G boundary for compatibility with some processors.) This means that a 64bit PV dom0 ends up occupying the highest addresses in RAM. Naturally, this is a problem for devices which can't do 64bit DMA. The xen-swiotlb is a region (default 128M iirc) which dom0 requests of Xen during startup. It is below the 4G boundary, and physically contiguous (unlike most of the rest of PV guest memory). Hence it's a suitable bounce buffer for any 32-bit-only devices. I don't know exactly how this translates to Linux internals, but most devices are fine and it's routinely the mpt2/3sas drivers which encounter problems. It would be lovely if we could get to the bottom of this for once and for all. ~Andrew
