> On Feb 27, 2023, at 1:18 PM, Peter Xu <peterx@xxxxxxxxxx> wrote: > > !! External Email > > On Thu, Feb 23, 2023 at 05:11:11PM +0000, Nadav Amit wrote: >> From my experience with UFFD, proper ordering of events is crucial, although it >> is not always done well. Therefore, we should aim for improvement, not >> regression. I believe that utilizing the pagemap-based mechanism for WP'ing >> might be a step in the wrong direction. I think that it would have been better >> to emit a 'UFFD_FEATURE_WP_ASYNC' WP-log (and ordered) with UFFD #PF and >> events. The 'UFFD_FEATURE_WP_ASYNC'-log may not need to wake waiters on the >> file descriptor unless the log is full. > > Yes this is an interesting question to think about.. > > Keeping the data in the pgtable has one good thing that it doesn't need any > complexity on maintaining the log, and no possibility of "log full". I understand your concern, but I think that eventually it might be simpler to maintain, since the logic of how to process the log is moved to userspace. At the same time, handling inputs from pagemap and uffd handlers and sync’ing them would not be too easy for userspace. But yes, allocation on the heap for userfaultfd_wait_queue-like entries would be needed, and there are some issues of ordering the events (I think all #PF and other events should be ordered regardless) and how not to traverse all async-userfaultfd_wait_queue’s (except those that block if the log is full) when a wakeup is needed. > > If there's possible "log full" then the next question is whether we should > let the worker wait the monitor if the monitor is not fast enough to > collect those data. It adds some slight dependency on the two threads, I > think it can make the tracking harder or impossible in latency sensitive > workloads. Again, I understand your concern. But this model that I propose is not new. It is used with PML (page-modification logging) and KVM, and IIRC there is a similar interface between KVM and QEMU to provide this information. There are endless other examples for similar producer-consumer mechanisms that might lead to stall in extreme cases. > > The other thing is we can also make the log "never gonna full" by making it > a bitmap covering any registered ranges, but I don't either know whether > it'll be worth it for the effort. I do not see a benefit of half-log half-scan. It tries to take the data-structure of one format and combine it with another. Anyhow, I was just giving my 2 cents. Admittedly, I did not follow the threads of previous versions and I did not see userspace components that use the API to say something smart. Personally, I do not find the current API proposal to be very consistent and simple, and it seems to me that it lets pagemap do userfaultfd-related tasks, which might be considered inappropriate and non-intuitive. If I derailed the discussion, I apologize.
