On Thu, Oct 10, 2024 at 1:56 PM Andrii Nakryiko <andrii@xxxxxxxxxx> wrote: > > Implement speculative (lockless) resolution of VMA to inode to uprobe, > bypassing the need to take mmap_lock for reads, if possible. Patch #1 by Suren > adds mm_struct helpers that help detect whether mm_struct was changed, which > is used by uprobe logic to validate that speculative results can be trusted > after all the lookup logic results in a valid uprobe instance. Patch #2 > follows to make mm_lock_seq into 64-bit counter (on 64-bit architectures), as > requested by Jann Horn. > > Patch #3 is a simplification to uprobe VMA flag checking, suggested by Oleg. > > And, finally, patch #4 is the speculative VMA-to-uprobe resolution logic > itself, and is the focal point of this patch set. It makes entry uprobes in > common case scale very well with number of CPUs, as we avoid any locking or > cache line bouncing between CPUs. See corresponding patch for details and > benchmarking results. > > Note, this patch set assumes that FMODE_BACKING files were switched to have > SLAB_TYPE_SAFE_BY_RCU semantics, which was recently done by Christian Brauner > in [0]. This change can be pulled into perf/core through stable > tags/vfs-6.13.for-bpf.file tag from [1]. > > [0] https://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git/commit/?h=vfs-6.13.for-bpf.file&id=8b1bc2590af61129b82a189e9dc7c2804c34400e > [1] git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs.git > > v2->v3: > - dropped kfree_rcu() patch (Christian); > - added data_race() annotations for fields of vma and vma->vm_file which could > be modified during speculative lookup (Oleg); > - fixed int->long problem in stubs for mmap_lock_speculation_{start,end}(), > caught by Kernel test robot; > v1->v2: > - adjusted vma_end_write_all() comment to point out it should never be called > manually now, but I wasn't sure how ACQUIRE/RELEASE comments should be > reworded (previously requested by Jann), so I'd appreciate some help there > (Jann); > - int -> long change for mm_lock_seq, as agreed at LPC2024 (Jann, Suren, Liam); > - kfree_rcu_mightsleep() for FMODE_BACKING (Suren, Christian); > - vm_flags simplification in find_active_uprobe_rcu() and > find_active_uprobe_speculative() (Oleg); > - guard(rcu)() simplified find_active_uprobe_speculative() implementation. > > Andrii Nakryiko (3): > mm: switch to 64-bit mm_lock_seq/vm_lock_seq on 64-bit architectures > uprobes: simplify find_active_uprobe_rcu() VMA checks > uprobes: add speculative lockless VMA-to-inode-to-uprobe resolution > > Suren Baghdasaryan (1): > mm: introduce mmap_lock_speculation_{start|end} > > include/linux/mm.h | 6 ++-- > include/linux/mm_types.h | 7 ++-- > include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++------- > kernel/events/uprobes.c | 52 +++++++++++++++++++++++++++- > kernel/fork.c | 3 -- > 5 files changed, 119 insertions(+), 21 deletions(-) > > -- > 2.43.5 > This applies cleanly to tip/perf/core with or without Jiri's patches ([0]). No need to rebase and resend, this is ready to go in. [0] https://lore.kernel.org/all/20241018202252.693462-1-jolsa@xxxxxxxxxx/
