On Thu, 15 Jul 2021 10:44:32 +0100, Qu Wenruo <wqu@xxxxxxxx> wrote: > > > > On 2021/7/15 下午5:28, Robin Murphy wrote: > > On 2021-07-15 09:55, Qu Wenruo wrote: > >> Hi, > >> > >> Recently I'm playing around the Nvidia Xavier AGX board, which has > >> VHE extension support. > >> > >> In theory, considering the CPU and memory, it should be pretty > >> powerful compared to boards like RPI CM4. > >> > >> But to my surprise, KVM runs pretty poor on Xavier. > >> > >> Just booting the edk2 firmware could take over 10s, and 20s to > >> fully boot the kernel. > >> Even my VM on RPI CM4 has way faster boot time, even just running > >> on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core. > >> > >> This is definitely out of my expectation, I double checked to be > >> sure that it's running in KVM mode. > >> > >> But further digging shows that, since Xavier AGX CPU supports VHE, > >> kvm is running in VHE mode other than HYP mode on CM4. > >> > >> Is there anyway to manually disable VHE mode to test the more > >> common HYP mode on Xavier? > > > > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its > > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should > > do that. > > Thanks for this one, I stupidly only searched modinfo of kvm, and > didn't even bother to search arch/arm64/kvm... > > > > > However I'd imagine the discrepancy is likely to be something more > > fundamental to the wildly different microarchitectures. There's > > certainly no harm in giving non-VHE a go for comparison, but I > > wouldn't be surprised if it turns out even slower... > > You're totally right, with nvhe mode, it's still the same slow speed. My experience with Denver (Nvidia's previous core) is that it is horribly slow when running KVM. I guess that the JIT-like microarch fares poorly with exceptions and save-restore operations. > BTW, what did you mean by the "wildly different microarch"? > Is ARMv8.2 arch that different from ARMv8 of RPI4? > > And any extra methods I could try to explore the reason of the slowness? > > At least RPI CM4 is beyond my expectation and is working pretty fine. > > Thanks, > Qu > > > > > Robin. > > > >> BTW, this is the dmesg related to KVM on Xavier, running v5.13 > >> upstream kernel, with 64K page size: > >> [ 0.852357] kvm [1]: IPA Size Limit: 40 bits > >> [ 0.857378] kvm [1]: vgic interrupt IRQ9 > >> [ 0.862122] kvm: pmu event creation failed -2 And this isn't going to help finding out the bottleneck, as the kernel doesn't find a PMU. On Denver, once the PMU is enabled, profiling anything makes the whole thing even slower. At which point, I just parked the board and forgot about it. Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
