On 05/05/20 13:28, Kashyap Chamarthy wrote: > + > +Live migration with nested KVM > +------------------------------ > + > +The below live migration scenarios should work as of Linux kernel 5.3 > +and QEMU 4.2.0 for x86; for s390x, even older versions might work. > +In all the below cases, L1 exposes ``/dev/kvm`` in it, i.e. the L2 guest > +is a "KVM-accelerated guest", not a "plain emulated guest" (as done by > +QEMU's TCG). > + > +- Migrating a nested guest (L2) to another L1 guest on the *same* bare > + metal host. > + > +- Migrating a nested guest (L2) to another L1 guest on a *different* > + bare metal host. > + > +- Migrating an L1 guest, with an *offline* nested guest in it, to > + another bare metal host. > + > +- Migrating an L1 guest, with a *live* nested guest in it, to another > + bare metal host. > + > +Limitations on Linux kernel versions older than 5.3 (x86) > +--------------------------------------------------------- > + > +On Linux kernel versions older than 5.3, once an L1 guest has started an > +L2 guest, the L1 guest would no longer capable of being migrated, saved, > +or loaded (refer to QEMU documentation on "save"/"load") until the L2 > +guest shuts down. > + > +Attempting to migrate or save-and-load an L1 guest while an L2 guest is > +running will result in undefined behavior. You might see a ``kernel > +BUG!`` entry in ``dmesg``, a kernel 'oops', or an outright kernel panic. > +Such a migrated or loaded L1 guest can no longer be considered stable or > +secure, and must be restarted. > + > +Migrating an L1 guest merely configured to support nesting, while not > +actually running L2 guests, is expected to function normally. > +Live-migrating an L2 guest from one L1 guest to another is also expected > +to succeed. > + This is a bit optimistic, as AMD is not supported yet. Please review the following incremental patch: diff --git a/Documentation/virt/kvm/running-nested-guests.rst b/Documentation/virt/kvm/running-nested-guests.rst --- a/Documentation/virt/kvm/running-nested-guests.rst +++ b/Documentation/virt/kvm/running-nested-guests.rst @@ -182,11 +182,23 @@ Enabling "nested" (s390x) Live migration with nested KVM ------------------------------ -The below live migration scenarios should work as of Linux kernel 5.3 -and QEMU 4.2.0 for x86; for s390x, even older versions might work. -In all the below cases, L1 exposes ``/dev/kvm`` in it, i.e. the L2 guest -is a "KVM-accelerated guest", not a "plain emulated guest" (as done by -QEMU's TCG). +Migrating an L1 guest, with a *live* nested guest in it, to another +bare metal host, works as of Linux kernel 5.3 and QEMU 4.2.0 for +Intel x86 systems, and even on older versions for s390x. + +On AMD systems, once an L1 guest has started an L2 guest, the L1 guest +should no longer be migrated or saved (refer to QEMU documentation on +"savevm"/"loadvm") until the L2 guest shuts down. Attempting to migrate +or save-and-load an L1 guest while an L2 guest is running will result in +undefined behavior. You might see a ``kernel BUG!`` entry in ``dmesg``, a +kernel 'oops', or an outright kernel panic. Such a migrated or loaded L1 +guest can no longer be considered stable or secure, and must be restarted. +Migrating an L1 guest merely configured to support nesting, while not +actually running L2 guests, is expected to function normally even on AMD +systems but may fail once guests are started. + +Migrating an L2 guest is expected to succeed, so all the following +scenarios should work even on AMD systems: - Migrating a nested guest (L2) to another L1 guest on the *same* bare metal host. @@ -194,30 +206,7 @@ QEMU's TCG). - Migrating a nested guest (L2) to another L1 guest on a *different* bare metal host. -- Migrating an L1 guest, with an *offline* nested guest in it, to - another bare metal host. - -- Migrating an L1 guest, with a *live* nested guest in it, to another - bare metal host. - -Limitations on Linux kernel versions older than 5.3 (x86) ---------------------------------------------------------- - -On Linux kernel versions older than 5.3, once an L1 guest has started an -L2 guest, the L1 guest would no longer capable of being migrated, saved, -or loaded (refer to QEMU documentation on "save"/"load") until the L2 -guest shuts down. - -Attempting to migrate or save-and-load an L1 guest while an L2 guest is -running will result in undefined behavior. You might see a ``kernel -BUG!`` entry in ``dmesg``, a kernel 'oops', or an outright kernel panic. -Such a migrated or loaded L1 guest can no longer be considered stable or -secure, and must be restarted. - -Migrating an L1 guest merely configured to support nesting, while not -actually running L2 guests, is expected to function normally. -Live-migrating an L2 guest from one L1 guest to another is also expected -to succeed. +- Migrating a nested guest (L2) to a bare metal host. Reporting bugs from nested setups -----------------------------------
