Hi Keqian,
On 2020-07-06 02:28, zhukeqian wrote:
Hi Catalin and Marc,
On 2020/7/2 21:55, Keqian Zhu wrote:
This patch series add support for dirty log based on HW DBM.
It works well under some migration test cases, including VM with 4K
pages or 2M THP. I checked the SHA256 hash digest of all memory and
they keep same for source VM and destination VM, which means no dirty
pages is missed under hardware DBM.
Some key points:
1. Only support hardware updates of dirty status for PTEs. PMDs and
PUDs
are not involved for now.
2. About *performance*: In RFC patch, I have mentioned that for every
64GB
memory, KVM consumes about 40ms to scan all PTEs to collect dirty
log.
This patch solves this problem through two ways: HW/SW dynamic
switch
and Multi-core offload.
HW/SW dynamic switch: Give userspace right to enable/disable hw
dirty
log. This adds a new KVM cap named KVM_CAP_ARM_HW_DIRTY_LOG. We can
achieve this by change the kvm->arch.vtcr value and kick vCPUs out
to
reload this value to VCTR_EL2. Then userspace can enable hw dirty
log
at the begining and disable it when dirty pages is little and about
to
stop VM, so VM downtime is not affected.
Multi-core offload: Offload the PT scanning workload to multi-core
can
greatly reduce scanning time. To promise we can complete in time, I
use
smp_call_fuction to realize this policy, which utilize IPI to
dispatch
workload to other CPUs. Under 128U Kunpeng 920 platform, it just
takes
about 5ms to scan PTs of 256 RAM (use mempress and almost all PTs
have
been established). And We dispatch workload iterately (every CPU
just
scan PTs of 512M RAM for each iteration), so it won't affect
physical
CPUs seriously.
What do you think of these two methods to solve high-cost PTs scaning?
Maybe
you are waiting for PML like feature on ARM :-) , but for my test, DBM
is usable
after these two methods applied.
Useable, maybe. But leaving to userspace the decision to switch from one
mode to another isn't an acceptable outcome. Userspace doesn't need nor
want to know about this.
Another thing is that sending IPIs all over to trigger scanning may
work well on a system that runs a limited number of guests (or some
other userspace, actually), but I seriously doubt that it is impact
free once you start doing this on an otherwise loaded system.
You may have better results by having an alternative mapping of your
S2 page tables so that they are accessible linearly, which would
sidestep the PT parsing altogether, probably saving some cycles. But
this is still a marginal gain compared to the overall overhead of
scanning 4kB of memory per 2MB of guest RAM, as opposed to 64 *bytes*
per 2MB (assuming strict 4kB mappings at S2, no block mappings).
Finally, this doesn't work with pages dirtied from DMA, which is the
biggest problem. If you cannot track pages that are dirtied behind your
back, what is the purpose of scanning the dirty bits?
As for a PML-like feature, this would only be useful if the SMMU
architecture took part in it and provided consistent logging of
the dirtied pages in the IPA space. Only having it at the CPU level
would be making the exact same mistake.
So, given the above, I remain unconvinced by this approach.
Thanks,
M.
--
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@xxxxxxxxxxxxxxxxxxxxx
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
