On Fri, 28 Jul 2023 at 12:39, Peter Xu <peterx@xxxxxxxxxx> wrote:
>
> But then does it means that any gup-only user will have numa balancing
> completely disabled?
Why would we ever care about a GUP-only user?
Who knows where the actual access is coming from? It might be some
device that is on a different node entirely.
And even if the access is local from the CPU, it
(a) might have happened after we moved somewhere else
(b) who cares about the extra possible NUMA overhead when we just
wasted *thousands* of cycles on GUP?
So NUMA balancing really doesn't seem to make sense for GUP anyway as
far as I can see.
Now, the other side of the same thing is that (a) NUMA faulting should
be fairly rare and (b) once you do GUP, who cares anyway, so you can
also argue that "once you do GUP you might as well NUMA-fault, because
performance simply isn't an issue".
But I really think the real argument is "once you do GUP, numa
faulting is just crazy".
I think what happened is
- the GUP code couldn't tell NUMA and actual PROTNONE apart
- so the GUP code would punch through PROTNONE even when it shouldn't
- so people added FOLL_NUMA to say "I don't want you to punch
through, I want the NUMA fault"
- but then FOLL_FORCE ends up meaning that you actually *do* want to
punch through - regardless of NUMA or not - and now the two got tied
together, and we end up with nonsensical garbage like
if (!(gup_flags & FOLL_FORCE))
gup_flags |= FOLL_NUMA;
to say "oh, actually, to avoid punching through when we shouldn't,
we should NUMA fault".
so we ended up with that case where even if YOU DIDN'T CARE AT ALL,
you got FOLL_NUMA just so that you wouldn't punch through.
And now we're in the situation that we've confused FOLL_FORCE and
FOLL_NUMA, even though they have absolutely *nothing* to do with each
other, except for a random implementation detail about punching
through incorrectly that isn't even relevant any more.
I really think FOLL_NUMA should just go away. And that FOLL_FORCE
replacement for it is just wrong. If you *don't* do something without
FOLL_FORCE, you damn well shouldn't do it just because FOLL_FORCE is
set.
The *only* semantic meaning FOLL_FORCE should have is that it
overrides the vma protections for debuggers (in a very limited
manner). It should *not* affect any NUMA faulting logic in any way,
shape, or form.
Linus
