On 18.05.21 18:13, Claudio Imbrenda wrote:
On Tue, 18 May 2021 17:45:18 +0200
Christian Borntraeger <borntraeger@xxxxxxxxxx> wrote:
On 18.05.21 17:36, Claudio Imbrenda wrote:
On Tue, 18 May 2021 17:05:37 +0200
Cornelia Huck <cohuck@xxxxxxxxxx> wrote:
On Mon, 17 May 2021 22:07:47 +0200
Claudio Imbrenda <imbrenda@xxxxxxxxxxxxx> wrote:
Previously, when a protected VM was rebooted or when it was shut
down, its memory was made unprotected, and then the protected VM
itself was destroyed. Looping over the whole address space can
take some time, considering the overhead of the various
Ultravisor Calls (UVCs). This means that a reboot or a shutdown
would take a potentially long amount of time, depending on the
amount of used memory.
This patchseries implements a deferred destroy mechanism for
protected guests. When a protected guest is destroyed, its memory
is cleared in background, allowing the guest to restart or
terminate significantly faster than before.
There are 2 possibilities when a protected VM is torn down:
* it still has an address space associated (reboot case)
* it does not have an address space anymore (shutdown case)
For the reboot case, the reference count of the mm is increased,
and then a background thread is started to clean up. Once the
thread went through the whole address space, the protected VM is
actually destroyed.
For the shutdown case, a list of pages to be destroyed is formed
when the mm is torn down. Instead of just unmapping the pages when
the address space is being torn down, they are also set aside.
Later when KVM cleans up the VM, a thread is started to clean up
the pages from the list.
Just to make sure, 'clean up' includes doing uv calls?
yes
This means that the same address space can have memory belonging
to more than one protected guest, although only one will be
running, the others will in fact not even have any CPUs.
Are those set-aside-but-not-yet-cleaned-up pages still possibly
accessible in any way? I would assume that they only belong to the
in case of reboot: yes, they are still in the address space of the
guest, and can be swapped if needed
'zombie' guests, and any new or rebooted guest is a new entity that
needs to get new pages?
the rebooted guest (normal or secure) will re-use the same pages of
the old guest (before or after cleanup, which is the reason of
patches 3 and 4)
the KVM guest is not affected in case of reboot, so the userspace
address space is not touched.
Can too many not-yet-cleaned-up pages lead to a (temporary) memory
exhaustion?
in case of reboot, not much; the pages were in use are still in use
after the reboot, and they can be swapped.
in case of a shutdown, yes, because the pages are really taken aside
and cleared/destroyed in background. they cannot be swapped. they
are freed immediately as they are processed, to try to mitigate
memory exhaustion scenarios.
in the end, this patchseries is a tradeoff between speed and memory
consumption. the memory needs to be cleared up at some point, and
that requires time.
in cases where this might be an issue, I introduced a new KVM flag
to disable lazy destroy (patch 10)
Maybe we could piggy-back on the OOM-kill notifier and then fall back
to synchronous freeing for some pages?
I'm not sure I follow
once the pages have been set aside, it's too late
while the pages are being set aside, every now and then some memory
needs to be allocated. the allocation is atomic, not allowed to use
emergency reserves, and can fail without warning. if the allocation
fails, we clean up one page and continue, without setting aside
anything (patch 9)
so if the system is low on memory, the lazy destroy should not make the
situation too much worse.
the only issue here is starting a normal process in the host (maybe
a non secure guest) that uses a lot of memory very quickly, right after
a large secure guest has terminated.
I think page cache page allocations do not need to be atomic.
In that case the kernel might stil l decide to trigger the oom killer. We can
let it notify ourselves free 256 pages synchronously and avoid the oom kill.
Have a look at the virtio-balloon virtio_balloon_oom_notify
