On Fri, 22 Nov 2019 09:50:50 -0800 Daniel Verkamp <dverkamp@xxxxxxxxxxxx> wrote: > On Fri, Nov 22, 2019 at 7:29 AM Marc Zyngier <maz@xxxxxxxxxx> wrote: > > > > Hi Daniel, > > > > [please use my kernel.org address, as I'm not employed by ARM anymore] > > > > On 2019-11-21 20:16, Daniel Verkamp wrote: > > > Hi Marc, > > > > > > I noticed that in recent kernels (4.19 in our case), KVM no longer > > > works for 32-bit processes running on arm64 machines. We > > > (Crostini/crosvm, Linux VM support on Chromebooks) use this > > > KVM_COMPAT > > > on all arm64-based Chromebooks that support VMs, since our entire > > > userspace (including crosvm) is 32-bit. > > > > > > I found this commit (KVM: arm64: Prevent KVM_COMPAT from being > > > selected): > > > > > > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=37b65db85f9b2fc98267eee4a18d7506492e6e8c > > > > > > Is there any particular reason the compat ioctl is disabled (e.g. > > > actual bugs/breakage), or would it be possible to re-enable > > > KVM_COMPAT > > > for ARM64? > > > > Well, it never worked the first place! :( The whole API is entirely > > different, > > and you should feel extremely lucky that it ever seemed to work! > > I hadn't looked at this before, but it seems like the Rust bindings > for KVM we are using are actually generated from the aarch64 C headers > (e.g. struct kvm_regs contains sp_el1, etc. rather than svc_regs): > https://chromium.googlesource.com/chromiumos/platform/crosvm/+/37bd738b783053c2e94b4c215c32496bae5081ed/kvm_sys/src/aarch64/bindings.rs > So we are really passing the native 64-bit structs to the 32-bit > compat ioctl, which is probably bogus. Wow. It's amazing it did anything sensible! > I can see how this is unlikely to work in a normal C program compiled > against the 32-bit ARM uapi headers. Indeed. > > > In my brief testing with a 4.19-stable-based kernel on a new arm64 > > > board, everything seems to work fine with the commit above reverted. > > > > The key word here is *seems*. It may happen that the userspace ABI > > somehow align for some of the basic stuff, but it will fail for > > anything moderately serious (I really doubt QEMU would be able to > > run at all). > > > > I suppose to tried this with crosvm? 32 or 64bit guest? > > Yes, I am able to run the full Crostini environment using 32-bit > crosvm on an aarch64 Chromebook kernel based on 4.19.84 stable. The > guest kernel and userspace is 64 bit in this case (we don't support > 32-bit guest). > > I also just tried a 32-bit build of kvmtool (lkvm), since we have a > convenient ebuild to cross-compile it in the Chrome OS SDK, but I > wasn't able to get it to boot (it fails at a call to KVM_GET_ONE_REG). > I didn't attempt to debug it, but I'm assuming it's due to the ABI > mismatch you mentioned. That's what I was expecting indeed. The registers are identified differently to make my life easier (and at some point I considered writing a compat interface before making up my mind and canning the thing altogether). > It sounds like disabling the compat ioctl is probably the right thing > to do in this case; I'll have to see if we can build crosvm as a > 64-bit executable instead in the future (not sure how easily this will > be given the way the Chromium OS build system works). You may have to end-up with a static binary, which is not very appealing... M. -- Jazz is not dead. It just smells funny... _______________________________________________ kvmarm mailing list kvmarm@xxxxxxxxxxxxxxxxxxxxx https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
