> > > Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines. > > > > Does the host machine have 2048 CPUs (or more) as well in your usecase? No, the largest system that I've used had 1792 CPUs. > > I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS > > based on NR_CPUS. That way KVM can scale up on large machines without > > using more memory on small machines. > > > > e.g. > > > > /* Provide backwards compatibility. */ > > #if NR_CPUS < 1024 > > #define KVM_MAX_VCPUS 1024 > > #else > > #define KVM_MAX_VCPUS NR_CPUS > > #endif > > > > The only downside I can see for this approach is if you are trying to > > kick the tires a new large VM on a smaller host because the new "large > > host" hardware hasn't landed yet. > > FWIW, while I don't think there's anything wrong with such approach, it > won't help much distro kernels which are not recompiled to meet the > needs of a particular host. According to Kyle's numbers, the biggest > growth is observed with 'struct kvm_ioapic' and that's only because of > 'struct rtc_status' embedded in it. Maybe it's possible to use something > different from a KVM_MAX_VCPU_IDS-bound flat bitmask there? I'm not sure > how important this is as it's just another 4K per-VM and when guest's > memory is taken into account it's probably not much. > > The growth in 'struct kvm'/'struct kvm_arch' seems to be insignificant > and on-stack allocations are probably OK. If NR_CPUS is used, KVM_MAX_VCPUS will be 8192 and KVM_MAX_VCPU_IDS will be 32768 when MAXSMP is set. Thanks, Kyle Meyer ________________________________________ From: Vitaly Kuznetsov <vkuznets@xxxxxxxxxx> Sent: Tuesday, June 14, 2022 3:27 AM To: David Matlack; Meyer, Kyle Cc: kvm list; X86 ML; Borislav Petkov; Dave Hansen; Ingo Molnar; Thomas Gleixner; Anderson, Russ; Payton, Brian; H. Peter Anvin; Jim Mattson; Joerg Roedel; Sean Christopherson; Wanpeng Li Subject: Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048 David Matlack <dmatlack@xxxxxxxxxx> writes: > On Mon, Jun 13, 2022 at 11:35 AM Kyle Meyer <kyle.meyer@xxxxxxx> wrote: >> >> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines. > > Does the host machine have 2048 CPUs (or more) as well in your usecase? > > I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS > based on NR_CPUS. That way KVM can scale up on large machines without > using more memory on small machines. > > e.g. > > /* Provide backwards compatibility. */ > #if NR_CPUS < 1024 > #define KVM_MAX_VCPUS 1024 > #else > #define KVM_MAX_VCPUS NR_CPUS > #endif > > The only downside I can see for this approach is if you are trying to > kick the tires a new large VM on a smaller host because the new "large > host" hardware hasn't landed yet. FWIW, while I don't think there's anything wrong with such approach, it won't help much distro kernels which are not recompiled to meet the needs of a particular host. According to Kyle's numbers, the biggest growth is observed with 'struct kvm_ioapic' and that's only because of 'struct rtc_status' embedded in it. Maybe it's possible to use something different from a KVM_MAX_VCPU_IDS-bound flat bitmask there? I'm not sure how important this is as it's just another 4K per-VM and when guest's memory is taken into account it's probably not much. The growth in 'struct kvm'/'struct kvm_arch' seems to be insignificant and on-stack allocations are probably OK. -- Vitaly
