On Wed, May 22, 2024 at 15:59:36 +0800, d tbsky wrote:
> Hi:
>
> Jiri Denemark <jdenemar@xxxxxxxxxx>
> > Interesting, hypervisor-cpu-baseline is supposed to provide an
> > intersection of both CPU models. I wonder whether we have a bug
> > somewhere, could you share the host-model definition from virsh
> > domcapabilities from both hosts? The easiest way is running
> >
> > $ virsh domcapabilities | xmllint --xpath '//cpu/mode[@name="host-model"]' -
>
vmx-apicv-register,vmx-apicv-vid,vmx-posted-intr
> the result are below:
> for cpu i3-9100F:
>
> <mode name="host-model" supported="yes">
> <model fallback="forbid">Skylake-Client-IBRS</model>
> <vendor>Intel</vendor>
...
> </mode>
>
> for cpu E3-1280 V3:
>
> <mode name="host-model" supported="yes">
> <model fallback="forbid">Haswell-noTSX-IBRS</model>
> <vendor>Intel</vendor>
...
> </mode>
Thanks, neither CPU model contains any of the features migration is
later complaining about (vmx-apicv-register, vmx-apicv-vid,
vmx-posted-intr). But they are all included in the Haswell-noTSX-IBRS
model while missing in Skylake-Client-IBRS. So I guess if you use the
baseline CPU model and start a domain using it on E3-1280 V3 the
features might be enabled by QEMU (because the baseline would not
explicitly disable them) and thus migrating to i3-9100F would result in
such error. Technically the error is correct as the guest could expect
the features to be present and try to use them after migration, which
would likely cause the guest to crash. On the other hand the computed
baseline CPU model would be wrong if my theory is correct.
So could you also provide
(1) the output of "virsh hypervisor-cpu-baseline" and
(2) the <cpu> element from the domain XML before it is started
(3) and once it is running on the host from which migration fails?
An easy way to extract the <cpu> element from domain XML is:
virsh dumpxml nest | xmllint --xpath '/domain/cpu' -
> > is it ok the drop all these vmx cpu features for a guest?
> > thanks a lot for help!
> >
> > Sure, unless you specifically need some of them, you can drop them. They
> > will be added to the XML anyway during VM startup.
That said, it may not be safe because of the reason I explained above,
but as long as it works, it can be used as a workaround.
> indeed. i try to run "virsh dumpxml" for a running vm and I saw many
> vmx lines.
> so I got confused. is "virsh dumpxml" supposed to be the same as
> "hypervisor-cpu-baseline" result?
Due to compatibility reasons libvirt's definition of specific CPU models
may differ from the definition QEMU uses so once a domain gets started
we query QEMU for the features that were actually enabled (as the set
may differ from what we asked for) and updates the domain XML
accordingly. Ideally when using a CPU definition computed by
hypervisor-cpu-baseline there should be no features added in domain XML
(virsh dumpxml) once the domain starts. In you case (as long as my
theory is correct) features that we have in our definition of
Haswell-noTSX-IBRS are added once domain starts and those features are
not available on the second host, which should never happen as such
features should have been explicitly disabled in the baseline CPU.
> PS: according to redhat
> document(https://access.redhat.com/documentation/zh-tw/red_hat_enterprise_linux/9/html/configuring_and_managing_virtualization/migrating-virtual-machines_configuring-and-managing-virtualization#verifying-host-cpu-compatibility-for-virtual-machine-migration_migrating-virtual-machines),
> it use "hypervisor-cpu-baseline" without "--migratable". it will bring
> in "invtsc" cpu feature. I can never live migrate with the feature.
Yes, without --migratable some features may be included in the result
that would block migration. They don't have to though, it depends the
common set of features among the hosts for which baseline is computed.
And invtsc is not the only non-migratable feature. In other words, using
--migratable for hypervisor-cpu-baseline is usually a good idea.
Jirka
