On Thu, Jul 15, 2021 at 11:00:42AM +0100, Robin Murphy wrote: > On 2021-07-15 10:44, Qu Wenruo wrote: > > > > > > On 2021/7/15 下午5:28, Robin Murphy wrote: > > > On 2021-07-15 09:55, Qu Wenruo wrote: > > > > Hi, > > > > > > > > Recently I'm playing around the Nvidia Xavier AGX board, which > > > > has VHE extension support. > > > > > > > > In theory, considering the CPU and memory, it should be pretty > > > > powerful compared to boards like RPI CM4. > > > > > > > > But to my surprise, KVM runs pretty poor on Xavier. > > > > > > > > Just booting the edk2 firmware could take over 10s, and 20s to > > > > fully boot the kernel. > > > > Even my VM on RPI CM4 has way faster boot time, even just > > > > running on PCIE2.0 x1 lane NVME, and just 4 2.1Ghz A72 core. > > > > > > > > This is definitely out of my expectation, I double checked to be > > > > sure that it's running in KVM mode. > > > > > > > > But further digging shows that, since Xavier AGX CPU supports > > > > VHE, kvm is running in VHE mode other than HYP mode on CM4. > > > > > > > > Is there anyway to manually disable VHE mode to test the more > > > > common HYP mode on Xavier? > > > > > > According to kernel-parameters.txt, "kvm-arm.mode=nvhe" (or its > > > low-level equivalent "id_aa64mmfr1.vh=0") on the command line should > > > do that. > > > > Thanks for this one, I stupidly only searched modinfo of kvm, and didn't > > even bother to search arch/arm64/kvm... > > > > > > > > However I'd imagine the discrepancy is likely to be something more > > > fundamental to the wildly different microarchitectures. There's > > > certainly no harm in giving non-VHE a go for comparison, but I > > > wouldn't be surprised if it turns out even slower... > > > > You're totally right, with nvhe mode, it's still the same slow speed. > > > > BTW, what did you mean by the "wildly different microarch"? > > Is ARMv8.2 arch that different from ARMv8 of RPI4? > > I don't mean Armv8.x architectural features, I mean the actual > implementation of NVIDIA's Carmel core is very, very different from > Cortex-A72 or indeed our newer v8.2 Cortex-A designs. > > > And any extra methods I could try to explore the reason of the slowness? > > I guess the first check would be whether you're trapping and exiting the VM > significantly more. I believe there are stats somewhere, but I don't know > exactly where, sorry - I know very little about actually *using* KVM :) > > If it's not that, then it might just be that EDK2 is doing a lot of cache > maintenance or system register modification or some other operation that > happens to be slower on Carmel compared to Cortex-A72. It would also be worthchecking tha the CPUs are running at the speed you expect, in e.g. case the lack of a DVFS driver means they're running slow, and this just happens to be more noticeable in a VM. You can estimate that by using `perf stat` on the host on a busy loop, and looking what the cpu-cycles count implies. Thanks, Mark.
