On Fri, Jul 22, 2022 at 4:41 PM Sean Christopherson <seanjc@xxxxxxxxxx> wrote: > > +x86 maintainers, patch 1 most definitely needs acceptance from folks beyond KVM. > > On Fri, Jul 22, 2022, Andrei Vagin wrote: > > Another option is the KVM platform. In this case, the Sentry (gVisor > > kernel) can run in a guest ring0 and create/manage multiple address > > spaces. Its performance is much better than the ptrace one, but it is > > still not great compared with the native performance. This change > > optimizes the most critical part, which is the syscall overhead. > > What exactly is the source of the syscall overhead, Here are perf traces for two cases: when "guest" syscalls are executed via hypercalls and when syscalls are executed by the user-space VMM: https://gist.github.com/avagin/f50a6d569440c9ae382281448c187f4e And here are two tests that I use to collect these traces: https://github.com/avagin/linux-task-diag/commit/4e19c7007bec6a15645025c337f2e85689b81f99 If we compare these traces, we can find that in the second case, we spend extra time in vmx_prepare_switch_to_guest, fpu_swap_kvm_fpstate, vcpu_put, syscall_exit_to_user_mode. > and what alternatives have been explored? Making arbitrary syscalls from > within KVM is mildly terrifying. "mildly terrifying" is a good sentence in this case:). If I were in your place, I would think about it similarly. I understand these concerns about calling syscalls from the KVM code, and this is why I hide this feature under a separate capability that can be enabled explicitly. We can think about restricting the list of system calls that this hypercall can execute. In the user-space changes for gVisor, we have a list of system calls that are not executed via this hypercall. For example, sigprocmask is never executed by this hypercall, because the kvm vcpu has its signal mask. Another example is the ioctl syscall, because it can be one of kvm ioctl-s. As for alternatives, we explored different ways: == Host Ring3/Guest ring0 mixed mode == This is how the gVisor KVM platform works right now. We don’t have a separate hypervisor, and the Sentry does its functions. The Sentry creates a KVM virtual machine instance, sets it up, and handles VMEXITs. As a result, the Sentry runs in the host ring3 and the guest ring0 and can transparently switch between these two contexts. When the Sentry starts, it creates a new kernel VM instance and maps its memory to the guest physical. Then it makes a set of page tables for the Sentry that mirrors the host virtual address space. When host and guest address spaces are identical, the Sentry can switch between these two contexts. The bluepill function switches the Sentry into guest ring0. It calls a privileged instruction (CLI) that is a no-op in the guest (by design, since we ensure interrupts are disabled for guest ring 0 execution) and triggers a signal on the host. The signal is handled by the bluepillHandler that takes a virtual CPU and executes it with the current thread state grabbed from a signal frame. As for regular VMs, user processes have their own address spaces (page tables) and run in guest ring3. So when the Sentry is going to execute a user process, it needs to be sure that it is running inside a VM, and it is the exact point when it calls bluepill(). Then it executes a user process with its page tables before it triggers an exception or a system call. All such events are trapped and handled in the Sentry. The Sentry is a normal Linux process that can trigger a fault and execute system calls. To handle these events, the Sentry returns to the host mode. If ring0 sysenter or exception entry point detects an event from the Sentry, they save the current thread state on a per-CPU structure and trigger VMEXIT. This returns us into bluepillHandler, where we set the thread state on a signal frame and exit from the signal handler, so the Sentry resumes from the point where it has been in the VM. In this scheme, the sentry syscall time is 3600ns. This is for the case when a system call is called from gr0. The benefit of this way is that only a first system call triggers vmexit and all subsequent syscalls are executed on the host natively. But it has downsides: * Each sentry system call trigger the full exit to hr3. * Each vmenter/vmexit requires to trigger a signal but it is expensive. * It doesn't allow to support Confidential Computing (SEV-ES/SGX). The Sentry has to be fully enclosed in a VM to be able to support these technologies. == Execute system calls from a user-space VMM == In this case, the Sentry is always running in VM, and a syscall handler in GR0 triggers vmexit to transfer control to VMM (user process that is running in hr3), VMM executes a required system call, and transfers control back to the Sentry. We can say that it implements the suggested hypercall in the user-space. The sentry syscall time is 2100ns in this case. The new hypercall does the same but without switching to the host ring 3. It reduces the sentry syscall time to 1000ns. == A new BPF hook to handle vmexit-s == https://github.com/avagin/linux-task-diag/commits/kvm-bpf This way allows us to reach the same performance numbers, but it gives less control over who and how use this functionality. Second, it requires adding a few questionable BPF helpers like calling syscall from BPF hooks. == Non-KVM platforms == We are experimenting with non-KVM platforms. We have the ptrace platform, but it is almost for experiments due to the slowness of the ptrace interface. Another idea was to add the process_vm_exec system call: https://lwn.net/Articles/852662/ This system call can significantly increase performance compared with the ptrace platform, but it is still slower than the KVM platform in its current form (without the new hypercall). But this is true only if we run the KVM platform on a bare-metal. In the case of nested-virtualization, the KVM platform becomes much slower, which is expected. We have another idea to use the seccomp notify to trap system calls, but it requires some kernel change to reach a reasonable performance. I am working on these changes and will present them soon. I want to emphasize that non-KVM platforms don't allow us to implement the confidential concept in gVisor, but this is one of our main goals concerning the KVM platform. All previous numbers have been getting from the same host (Xeon(R) Gold 6268CL, 5.19-rc5). > > > The idea of using vmcall to execute system calls isn’t new. Two large users > > of gVisor (Google and AntFinacial) have out-of-tree code to implement such > > hypercalls. > > > > In the Google kernel, we have a kvm-like subsystem designed especially > > for gVisor. This change is the first step of integrating it into the KVM > > code base and making it available to all Linux users. > > Can you please lay out the complete set of changes that you will be proposing? > Doesn't have to be gory details, but at a minimum there needs to be a high level > description that very clearly defines the scope of what changes you want to make > and what the end result will look like. > > It's practically impossible to review this series without first understanding the > bigger picture, e.g. if KVM_HC_HOST_SYSCALL is ultimately useless without the other > bits you plan to upstream, then merging it without a high level of confidence that > the other bits are acceptable is a bad idea since it commits KVM to supporting > unused ABI. I was not precise in my description. This is the only change that we need right now. The gVisor KVM platform is the real thing that exists today and works on the upstream kernels: https://cs.opensource.google/gvisor/gvisor/+/master:pkg/sentry/platform/kvm/ This hypercall improves its performance and makes it comparable with the google-internal platform. Thanks, Andrei
