On Mon, Feb 27, 2023 at 11:09:12PM +0000, Nadav Amit wrote: > > > > On Feb 27, 2023, at 1:18 PM, Peter Xu <peterx@xxxxxxxxxx> wrote: > > > > !! External Email > > > > On Thu, Feb 23, 2023 at 05:11:11PM +0000, Nadav Amit wrote: > >> From my experience with UFFD, proper ordering of events is crucial, although it > >> is not always done well. Therefore, we should aim for improvement, not > >> regression. I believe that utilizing the pagemap-based mechanism for WP'ing > >> might be a step in the wrong direction. I think that it would have been better > >> to emit a 'UFFD_FEATURE_WP_ASYNC' WP-log (and ordered) with UFFD #PF and > >> events. The 'UFFD_FEATURE_WP_ASYNC'-log may not need to wake waiters on the > >> file descriptor unless the log is full. > > > > Yes this is an interesting question to think about.. > > > > Keeping the data in the pgtable has one good thing that it doesn't need any > > complexity on maintaining the log, and no possibility of "log full". > > I understand your concern, but I think that eventually it might be simpler > to maintain, since the logic of how to process the log is moved to userspace. > > At the same time, handling inputs from pagemap and uffd handlers and sync’ing > them would not be too easy for userspace. I do not expect a common uffd-wp async user to provide a fault handler at all. In my imagination it's in most cases used standalone from other uffd modes; it means all the faults will still be handled by the kernel. Here we only leverage the accuracy of userfaultfd comparing to soft-dirty, so not really real "user"-faults. > > But yes, allocation on the heap for userfaultfd_wait_queue-like entries would > be needed, and there are some issues of ordering the events (I think all #PF > and other events should be ordered regardless) and how not to traverse all > async-userfaultfd_wait_queue’s (except those that block if the log is full) > when a wakeup is needed. Will there be an ordering requirement for an async mode? Considering it should be async to whatever else, I would think it's not a problem, but maybe I missed something. > > > > > If there's possible "log full" then the next question is whether we should > > let the worker wait the monitor if the monitor is not fast enough to > > collect those data. It adds some slight dependency on the two threads, I > > think it can make the tracking harder or impossible in latency sensitive > > workloads. > > Again, I understand your concern. But this model that I propose is not new. > It is used with PML (page-modification logging) and KVM, and IIRC there is > a similar interface between KVM and QEMU to provide this information. There > are endless other examples for similar producer-consumer mechanisms that > might lead to stall in extreme cases. Yes, I'm not against thinking of using similar structures here. It's just that it's definitely more complicated on the interface, at least we need yet one more interface to setup the rings and define its interfaces. Note that although Muhammud is defining another new interface here too for pagemap, I don't think it's strictly needed for uffd-wp async mode. One can use uffd-wp async mode with PM_UFFD_WP which is with current pagemap interface already. So what Muhammud is proposing here are two things to me: (1) uffd-wp async, plus (2) a new pagemap interface (which will closely work with (1) only if we need atomicity on get-dirty and reprotect). Defining new interface for uffd-wp async mode will be something extra, so IMHO besides the heap allocation on the rings, we need to also justify whether that is needed. That's why I think it's fine to go with what Muhammud proposed, because it's a minimum changeset at least for userfault to support an async mode, and anything else can be done on top if necessary. Going a bit back to the "lead to stall in extreme cases" above, just also want to mention that the VM use case is slightly different - dirty tracking is only heavily used during migration afaict, and it's a short period. Not a lot of people will complain performance degrades during that period because that's just rare. And, even without the ring the perf is really bad during migration anyway... Especially when huge pages are used to back the guest RAM. Here it's slightly different to me: it's about tracking dirty pages during any possible workload, and it can be monitored periodically and frequently. So IMHO stricter than a VM use case where migration is the only period to use it. > > > > > The other thing is we can also make the log "never gonna full" by making it > > a bitmap covering any registered ranges, but I don't either know whether > > it'll be worth it for the effort. > > I do not see a benefit of half-log half-scan. It tries to take the > data-structure of one format and combine it with another. What I'm saying here is not half-log / half-scan, but use a single bitmap to store what page is dirty, just like KVM_GET_DIRTY_LOG. I think it avoids any above "stall" issue. > > Anyhow, I was just giving my 2 cents. Admittedly, I did not follow the > threads of previous versions and I did not see userspace components that > use the API to say something smart. Actually similar here. :) So I'm probably not the best one to describe what is the best to look as API. What I know is I think the new pagemap interface is welcomed by CRIU developers, so it may be something good with/without userfaultfd getting involved already. I see this as "let's add one more bit for uffd-wp" in the new interface only. Quotting some link I got from Muhammud before with CRIU usage: https://lore.kernel.org/all/YyiDg79flhWoMDZB@xxxxxxxxx https://lore.kernel.org/all/20221014134802.1361436-1-mdanylo@xxxxxxxxxx > Personally, I do not find the current API proposal to be very consistent > and simple, and it seems to me that it lets pagemap do > userfaultfd-related tasks, which might be considered inappropriate and > non-intuitive. Yes, I agree. I just don't know what's the best way to avoid this. The issue here IIUC is Muhammud needs one operation to do what Windows does with getWriteWatch() API. It means we need to mix up GET and PROTECT in a single shot. If we want to use pagemap as GET, then no choice to PROTECT also here to me. I think it'll be the same to soft-dirty if it's used, it means we'll extend soft-dirty modifications from clear_refs to pagemap too which I also don't think it's as clean. > > If I derailed the discussion, I apologize. Not at all. I just wished you joined earlier! -- Peter Xu
