On Mon, Oct 02, 2023, Michael Roth wrote: > On Thu, Sep 28, 2023 at 11:31:51AM -0700, Sean Christopherson wrote: > > On Fri, Sep 22, 2023, Michael Roth wrote: > > > On Thu, Sep 21, 2023 at 01:33:23PM -0700, Sean Christopherson wrote: > > > > + /* > > > > + * For simplicity, allow mapping a hugepage if and only if the entire > > > > + * binding is compatible, i.e. don't bother supporting mapping interior > > > > + * sub-ranges with hugepages (unless userspace comes up with a *really* > > > > + * strong use case for needing hugepages within unaligned bindings). > > > > + */ > > > > + if (!IS_ALIGNED(slot->gmem.pgoff, 1ull << *max_order) || > > > > + !IS_ALIGNED(slot->npages, 1ull << *max_order)) > > > > + *max_order = 0; > > > > > > Thanks for working this in. Unfortunately on x86 the bulk of guest memory > > > ends up getting slotted directly above legacy regions at GFN 0x100, > > > > Can you provide an example? I'm struggling to understand what the layout actually > > is. I don't think it changes the story for the kernel, but it sounds like there > > might be room for optimization in QEMU? Or more likely, I just don't understand > > what you're saying :-) > > Here's one example, which seems to be fairly normal for an x86 boot: > > kvm_set_user_memory AddrSpace#0 Slot#0 flags=0x4 gpa=0x0 size=0x80000000 ua=0x7f24afc00000 ret=0 restricted_fd=19 restricted_offset=0x0 > ^ QEMU creates Slot 0 for all of main guest RAM > kvm_set_user_memory AddrSpace#0 Slot#0 flags=0x0 gpa=0x0 size=0x0 ua=0x7f24afc00000 ret=0 restricted_fd=19 restricted_offset=0x0 > kvm_set_user_memory AddrSpace#0 Slot#0 flags=0x4 gpa=0x0 size=0xc0000 ua=0x7f24afc00000 ret=0 restricted_fd=19 restricted_offset=0x0 > kvm_set_user_memory AddrSpace#0 Slot#3 flags=0x6 gpa=0xc0000 size=0x20000 ua=0x7f2575000000 ret=0 restricted_fd=33 restricted_offset=0x0 > kvm_set_user_memory AddrSpace#0 Slot#4 flags=0x6 gpa=0xe0000 size=0x20000 ua=0x7f2575400000 ret=0 restricted_fd=31 restricted_offset=0x0 > ^ legacy regions are created and mapped on top of GPA ranges [0xc0000:0xe0000) and [0xe0000:0x100000) > kvm_set_user_memory AddrSpace#0 Slot#5 flags=0x4 gpa=0x100000 size=0x7ff00000 ua=0x7f24afd00000 ret=0 restricted_fd=19 restricted_offset=0x100000 > ^ QEMU divides Slot 0 into Slot 0 at [0x0:0xc0000) and Slot 5 at [0x100000:0x80000000) > Both Slots still share the same backing memory allocation, so same gmem > fd 19 is used,but Slot 5 is assigned to offset 0x100000, whih is not > 2M-aligned > > I tried messing with QEMU handling to pad out guest_memfd offsets to 2MB > boundaries but then the inode size needs to be enlarged to account for it > and things get a bit messy. Not sure if there are alternative approaches > that can be taken from userspace, but with normal malloc()'d or mmap()'d > backing memory the kernel can still allocate a 2MB backing page for the > [0x0:0x200000) range and I think KVM still handles that when setting up > NPT of sub-ranges so there might not be much room for further optimization > there. Oooh, duh. QEMU intentionally creates a gap for the VGA and/or BIOS holes, and so the lower DRAM chunk that goes from the end of the system reserved chunk to to TOLUD is started at an unaligned offset, even though 99% of the slot is properly aligned. Yeah, KVM definitely needs to support that. Requiring userspace to align based on the hugepage size could work, e.g. QEMU could divide slot 5 into N slots, to end up with a series of slots to get from 4KiB aligned => 2MiB aligned => 1GiB aligned. But pushing for that would be beyond stubborn. Thanks for being patient :-)
