Hi Eric, On 2019/8/22 17:08, Auger Eric wrote:
Hi Zenghui, On 8/13/19 10:50 AM, Zenghui Yu wrote:Hi folks, Since commit e25028c8ded0 ("KVM: arm/arm64: Bump VGIC_V3_MAX_CPUS to 512"), we seemed to be allowed to boot a 512U guest. But I failed to start it up with the latest QEMU. I guess there are at least *two* reasons (limitations). First I got a QEMU abort: "kvm_set_irq: Invalid argument" Enable the trace_kvm_irq_line() under debugfs, when it comed with vcpu-256, I got: "Inject UNKNOWN interrupt (3), vcpu->idx: 0, num: 23, level: 0" and kvm_vm_ioctl_irq_line() returns -EINVAL to user-space... So the thing is that we only have 8 bits for vcpu_index field ([23:16]) in KVM_IRQ_LINE ioctl. irq_type field will be corrupted if we inject a PPI to vcpu-256, whose vcpu_index will take 9 bits. I temporarily patched the KVM and QEMU with the following diff: ---8<--- diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 95516a4..39a0fb1 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -325,10 +325,10 @@ struct kvm_vcpu_events { #define KVM_ARM_VCPU_TIMER_IRQ_PTIMER 1 /* KVM_IRQ_LINE irq field index values */ -#define KVM_ARM_IRQ_TYPE_SHIFT 24 -#define KVM_ARM_IRQ_TYPE_MASK 0xff +#define KVM_ARM_IRQ_TYPE_SHIFT 28 +#define KVM_ARM_IRQ_TYPE_MASK 0xf #define KVM_ARM_IRQ_VCPU_SHIFT 16 -#define KVM_ARM_IRQ_VCPU_MASK 0xff +#define KVM_ARM_IRQ_VCPU_MASK 0xfff #define KVM_ARM_IRQ_NUM_SHIFT 0 #define KVM_ARM_IRQ_NUM_MASK 0xffff ---8<--- It makes things a bit better, it also immediately BREAKs the api with old versions. Next comes one more QEMU abort (with the "fix" above): "Failed to set device address: No space left on device" We register two io devices (rd_dev and sgi_dev) on KVM_MMIO_BUS for each redistributor. 512 vcpus take 1024 io devices, which is beyond the maximum limitation of the current kernel - NR_IOBUS_DEVS (1000). So we get a ENOSPC error here.Do you plan to send a patch for increasing the NR_IOBUS_DEVS? Otherwise I can do it.
[...]
You mentioned you started working on the QEMU fix. Do you plan to submit patches to the ML or do you want me to attempt to fix the situation on userspace?
Actually, I haven't started anything yet. It would be great if you can help to fix this in QEMU. Yes, please help. :) And please cc me when you send the patch so that I can do some tests in the free time. Many thanks, zenghui _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
