On Wed, Aug 29, 2012 at 10:49:04AM -0300, Eduardo Habkost wrote: > > Normally CPUID will tell you if such important MSR is available. > > So we can check that at destination. > > How can qemu check it, if when the qemu code was written when the MSR > didn't even exist yet? > > (You could add an interface to check for that, but there's no KVM > ioctl() to tell qemu "given these CPUID bits, can I safely drop this MSR > that I don't even know about?") > So this is what I suggest exactly. Add a new ioctl like that. > > > > > > > > > > > > > > > > > > > > On the other hand, a mode of operation that doesn't require updating > > > > > > > > QEMU every time there's a new bit of guest-visible state to be migrated > > > > > > > > would be nice (just like the "-cpu host" mode, that doesn't require > > > > > > > > updating QEMU for every new CPU feature, is nice for some use cases). I > > > > > > > > just don't know how to make work with the current migration protocol. > > > > > > > > > > > > > > > > > > > > > > I don't understand. What is the problem with the proposal? > > > > > > > What will not work with our protocol? > > > > > > > Can you give an example please? > > > > > > > > > > Yes: > > > > > > > > > > Suppose kernel 3.7 introduces KVM_FOO_MSR and CPUID_KVM_FOO. > > > > > > > > > > Also, suppose QEMU 1.2 doesn't know anything about KVM_FOO, because it > > > > > was release before this feature was introduced. > > > > > > > > > > > > > > > Then you run "qemu-1.2 -M pc-1.2" on a 3.7 host kernel. qemu-1.2 can do > > > > > two things here: > > > > > > > > > > 1) Not enable CPUID_KVM_FOO > > > > > > > > > > In this case, the guest should not use KVM_FOO_MSR and the MSR does not > > > > > need to be migrated (the guest may try to use it, but the behavior when > > > > > trying to use it is undefined). Sending the MSR value when migrating > > > > > would be useless. > > > > > > > > > > > > > > > 2) Enable CPUID_KVM_FOO. > > > > > > > > > > In this case, the guest may try to use the feature and write something > > > > > into KVM_FOO_MSR. Sending the MSR value when migrating is absolutely > > > > > necessary > > > > > > > > > > --- > > > > > > > > > > Now assume you run "qemu-1.2 -M pc-1.2" in the destination host, running > > > > > the 3.6 kernel (without KVM_FOO). > > > > > > > > > > Then qemu-1.2 receives the KVM_FOO_MSR data in the migration stream. In > > > > > this case, qemu-1.2 simply can't decide if it's safe to drop the data > > > > > (and not tell KVM about it), or not. > > > > > If we simply send every MSR reported by the kernel, both the origin and > > > > > destination qemu-1.2 processes can't ever know if the MSR value is > > > > > important or not, because it doesn't know if it's part of the machine > > > > > state that has to be kept consistent. > > > > > We could introduce a mode of operation where _every_ MSR reported by KVM > > > > > is important and sent by the origin (and also where every MSR must be > > > > > handled by the destination, otherwise migration would fail). But this > > > > > new mode would break migration compatibility between two hosts running > > > > > the same machine-type, only because they are running different kernel > > > > > versions. But it may be useful for some use cases, so maybe it's > > > > > appropriate for a future "pc-next" machine-type (and for "-cpu host"), > > > > > but not for "pc-<version>". > > > > > > > > > > > > > In this example, we should migrate CPUID (don't we?). > > > > Destination should validate that CPUID supplied by source > > > > matches what it can support (doesn't it?). > > > > > > > > If we do and it does not, it's an unrelated bug: > > > > CPUID changing under guest's feet. > > > > > > CPUID changing under guest's feet is another problem, that we also have > > > to solve. > > > But we also have the problem of migration compatibility > > > between different host kernels. > > > > So here is the solution for both: on destination pass CPUID to kvm and > > it should come back unchanged. If it changed you fail migration. > > This doesn't solve the problem of having predictable migration > compatibility for "-M pc-<old-version>". > > The whole point of machine-types is to expose the "same machine" to the > guest, even if you change the hardware or host kernel. "qemu -M pc-1.x" > must expose the same machine configuration to the guest, it doesn 't > matter what's the host kernel version. I'd tend to disagree. The point is to make migration work and avoid things like windows re-activation trigger. Let's not be purists - many internal changes in qemu introduce subtle guest visible changes, if even in timing. > > > > > > > > Note that I am not saying that migrating all MSRs is wrong. It _is_ > > > good, as long as: > > > - The destination never ignores any of the incoming MSR values. > > > > What I am saying for MSRs added in last 2 years it is OK to ignore > > because CPUID check will tell you if it is supported > > and fail migration. > > Existing MSRs are easy to make work. The problem is about MSRs added to > the "msrs_to_save" list in the future. > > Also, the problem is not about being "safe" to ignore the MSR values, > it's about being "correct" (part of the expected behavior of the virtual > machine). The fact that most guests doen't crash when the virtual > machine doesn't behave as it should doesn't mean we should do it. > > Either the MSR is part of the machine state (and relevant to the guest), > or not. If it is relevant, it must be _always_ migrated and never > dropped by the destination. If it is not, it's useless to migrate it. > Yes. But it is better to keep all knowledge which is which in one place which is in kvm. > > > > > - We don't do that by default on "pc-<version>", or we defeat the > > > purpose of machine-types. > > > > > > I'll try to enumerate the problems I am trying to address (that are > > > related, but are separate problems): > > > > > > - MSR not being migrated when it should: > > > - Possible solution: migrate all MSRs even if qemu doesn't know what > > > they are. > > > - Constraint: migration destination must _never_ ignore any incoming > > > MSR value, because it can't decide if it is important to the guest > > > or not (with the current KVM interfaces). > > > - Problem with this solution: if we do that by default on > > > "pc-<version>", we break migration compatibility between hosts > > > with different kernel versions. > > > > Solution: add vcpu ioctl that tells you which MSRs to migrate > > (on source), depending on CPUID. > > This may be a solution for old-kernel => new-kernel migration, yes. But > this still doesn't solve the migration compatibility problem for > new-kernel => old-kernel migration (see below). > > > > > > > - Changing CPUID bits under guest's feet. > > > - Proposed solution: migrating CPUID bits, refuse migration if > > > destination doesn't support the same bits. > > > - It solves the compatibility problem for migration to a newer > > > kernel, but not to an older kernel. It helps to solve part of > > > the problem, but not all. > > > > How does not it save all of the problem? If destination kernel > > can support cpuid, then we are fine - it is new enough. > > See below. The problem is being able to migrate to an older host. > That's the whole point of machine-types! > > > > > > - Alternative solution: simply make the resulting CPUID bits not be a > > > function of the host kernel capabilities, but only of the qemu > > > configuration (except on "-cpu host" and "-M pc-next"). > > > > This perpetuates existing duplication of code between > > kvm and qemu. We are better off with logic in 1 place. > > Yes, it does, and I would love to avoid having the list inside QEMU, > too. But we can't avoid that because each machine-type defines a set of > available features/MSRs, so we have to have a machine-type => > list-of-features list somewhere, unfortunately. > Yes. But let us have machine type->cpuid list in qemu. kvm will have the cpuid->MSR logic. > > > > > - Migration compatibility/predictability: > > > - See my example above: feature introduced in a newer kernel, > > > migration to an older kernel. > > > > If it is enabled then migration fails. > > > > > - The only way I see to guarantee this is to never enable unknown > > > CPUID bits or MSRs. People who don't care about predictable > > > migration compatibility can use "-M pc-next", then. > > > > > > > Guarantee what? > > Guarantee that "-M pc-1.1" machines can be migrated to any host that is > already capable of running "-M pc-1.1". > I can't change the past. I am suggesting forward compatible approach so we'll be able to guarantee this for -M pc-1.2 and on. > > Just check dst can support all msrs and cpuid bits of src. > > Way to check is to ask kvm :) Not to add logic in qemu. > > Checking and making migration fail when it has to fail is not the > problem. The problem is that now "qemu -M pc-1.x" will result in a > different machine, depending on the host kernel version. This causes two > problems: > > - Now you don't know if your existing machine can be migrated to a > host running an older kernel (because now migration can fail even when > you are using the same machine-type on both sides). > - Different VMs using the same machine-type will get different machines > (with different sets of features), because they are running on > different kernel versions. > > This may be acceptable for "pc-next", but not for "pc-<version>". So you can whitelist CPUID bits. But leave MSRs alone, it is nasty enough with CPUID. -- MST -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@xxxxxxxxxxxxxxx More majordomo info at http://vger.kernel.org/majordomo-info.html
