Hi, Anish, On Wed, Apr 12, 2023 at 09:34:48PM +0000, Anish Moorthy wrote: > KVM's demand paging self test is extended to demonstrate the performance > benefits of using the two new capabilities to bypass the userfaultfd > wait queue. The performance samples below (rates in thousands of > pages/s, n = 5), were generated using [2] on an x86 machine with 256 > cores. > > vCPUs, Average Paging Rate (w/o new caps), Average Paging Rate (w/ new caps) > 1 150 340 > 2 191 477 > 4 210 809 > 8 155 1239 > 16 130 1595 > 32 108 2299 > 64 86 3482 > 128 62 4134 > 256 36 4012 The number looks very promising. Though.. > > [1] https://lore.kernel.org/linux-mm/CADrL8HVDB3u2EOhXHCrAgJNLwHkj2Lka1B_kkNb0dNwiWiAN_Q@xxxxxxxxxxxxxx/ > [2] ./demand_paging_test -b 64M -u MINOR -s shmem -a -v <n> -r <n> [-w] > A quick rundown of the new flags (also detailed in later commits) > -a registers all of guest memory to a single uffd. ... this is the worst case scenario. I'd say it's slightly unfair to compare by first introducing a bottleneck then compare with it. :) Jokes aside: I'd think it'll make more sense if such a performance solution will be measured on real systems showing real benefits, because so far it's still not convincing enough if it's only with the test especially with only one uffd. I don't remember whether I used to discuss this with James before, but.. I know that having multiple uffds in productions also means scattered guest memory and scattered VMAs all over the place. However split the guest large mem into at least a few (or even tens of) VMAs may still be something worth trying? Do you think that'll already solve some of the contentions on userfaultfd, either on the queue or else? With a bunch of VMAs and userfaultfds (paired with uffd fault handler threads, totally separate uffd queues), I'd expect to some extend other things can pop up already, e.g., the network bandwidth, without teaching each vcpu thread to report uffd faults themselves. These are my pure imaginations though, I think that's also why it'll be great if such a solution can be tested more or less on a real migration scenario to show its real benefits. Thanks, -- Peter Xu
