On Wed, Sep 22, 2021 at 10:43:40PM -0700, Jue Wang wrote: [...] > > > Could I know what's the workaround? Normally if the workaround works solidly, > > > then there's less need to introduce a kernel interface for that. Otherwise I'm > > > glad to look into such a formal proposal. > > > > The workaround is, for the region that you want to zap, run through > > this sequence of syscalls: mumap, mmap, and re-register with > > userfaultfd if it was registered before. If we're using tmpfs, we can > > use madvise(DONTNEED) instead, but this is kind of an abuse of the > > API. I don't think there's a guarantee that the PTEs will get zapped, > > but currently they will always get zapped if we're using tmpfs. I > > really like the idea of adding a new madvise() mode that is guaranteed > > to zap the PTEs. I see. > > > > > > > > > It's also useful for memory poisoning, I think, if the host > > > > decides some page(s) are "bad" and wants to intercept any future guest > > > > accesses to those page(s). > > > > > > Curious: isn't hwpoison information come from MCEs; or say, host kernel side? > > > Then I thought the host kernel will have full control of it already. > > > > > > Or there's other way that the host can try to detect some pages are going to be > > > rotten? So the userspace can do something before the kernel handles those > > > exceptions? > > > > Here's a general idea of how we would like to use userfaultfd to support MPR: > > > > If a guest accesses a poisoned page for the first time, we will get an > > MCE through the host kernel and send an MCE to the guest. The guest > > will now no longer be able to access this page, and we have to enforce > > this. After a live migration, the pages that were poisoned before > > probably won't still be poisoned (from the host's perspective), so we > > can't rely on the host kernel's MCE handling path. This is where > > userfaultfd and this new madvise mode come in: we can just > > madvise(MADV_ZAP) the poisoned page(s) on the target during a > > migration. Now all accesses will be routed to the VMM and we can > > inject an MCE. We don't *need* the new madvise mode, as we can also > > use fallocate(PUNCH_HOLE) (works for tmpfs and hugetlbfs), but it > > would be more convenient if we didn't have to use fallocate. > > > > Jue Wang can provide more context here, so I've cc'd him. There may be > > some things I'm wrong about, so Jue feel free to correct me. > > > James is right. > > The page is marked PG_HWPoison in the source VM host's kernel. The need > of intercepting guest accesses to it exist on the target VM host, where > the same physical page is no longer poisoned. > > On the target host, the hypervisor needs to intercept all guest accesses > to pages poisoned from the source VM host. Thanks for these information, James, Jue, Axel. I'm not familiar with memory failures yet, so please bare with me with a few naive questions. So now I can undertand that hw-poisonsed pages on src host do not mean these pages will be hw-poisoned on dest host too, but I may have missed the reason on why dest host needs to trap it with pgtable removed. AFAIU after pages got hw-poisoned on src, and after vmm injects MCEs into the guest, the guest shouldn't be accessing these pages any more, am I right? Then after migration completes, IIUC the guest shouldn't be accessing these pages too. My current understanding is, instead of trapping these pages on dest, we should just (somehow, which I have no real idea...) un-hw-poison these pages after migration because these pages are very possibly normal pages there. When there's real hw-poisoned pages reported on dst host, we should re-inject MCE errors to guest with another set of pages. Could you tell me where did I miss? Thanks, -- Peter Xu
