On Fri, Mar 01, 2024 at 09:01:07AM -0800, Alexei Starovoitov wrote: > On Fri, Mar 1, 2024 at 12:18 AM Jiri Olsa <olsajiri@xxxxxxxxx> wrote: > > > > On Thu, Feb 29, 2024 at 04:25:17PM -0800, Andrii Nakryiko wrote: > > > On Thu, Feb 29, 2024 at 6:39 AM Jiri Olsa <olsajiri@xxxxxxxxx> wrote: > > > > > > > > One of uprobe pain points is having slow execution that involves > > > > two traps in worst case scenario or single trap if the original > > > > instruction can be emulated. For return uprobes there's one extra > > > > trap on top of that. > > > > > > > > My current idea on how to make this faster is to follow the optimized > > > > kprobes and replace the normal uprobe trap instruction with jump to > > > > user space trampoline that: > > > > > > > > - executes syscall to call uprobe consumers callbacks > > > > > > Did you get a chance to measure relative performance of syscall vs > > > int3 interrupt handling? If not, do you think you'll be able to get > > > some numbers by the time the conference starts? This should inform the > > > decision whether it even makes sense to go through all the trouble. > > > > right, will do that > > I believe Yusheng measured syscall vs uprobe performance > difference during LPC. iirc it was something like 3x. > Certainly necessary to have a benchmark. > selftests/bpf/bench has one for uprobe. > Probably should extend with sys_bpf. > ok, did not know there was uprobe benchmark, will check > Regarding: > > replace the normal uprobe trap instruction with jump to > user space trampoline > > it should probably be a call to trampoline instead of a jump. > Unless you plan to generate a different trampoline for every location ? I wanted to store the ip of the uprobe as argument for the syscall, but the call instruction will push return address on stack and we can use it to get uprobe's address.. great > > Also how would you pick a space for a trampoline in the target process ? > Analyze /proc/pid/maps and look for gaps in executable sections? As Andrii mentioned in other response there's already one page mapped as '[uprobes]' mapping, it's used as trampoline for return uprobes (contains just int3 instruction) and as buffers to hold the original instruction for the single step execution I think if we endup with just single trampoline we can just use some of the space from that page, our trampoline should not be big > > We can start simple with a USDT that uses nop5 instead of nop1 > and explicit single trampoline for all USDT locations > that saves all (callee and caller saved) registers and > then does sys_bpf with a new cmd. ah, I did not realize USDTs are like that, will check, good idea > > To replace nop5 with a call to trampoline we can use text_poke_bp > approach: replace 1st byte with int3, replace 2-5 with target addr, > replace 1st byte to make an actual call insn. I'm bit in the dark in here, but uprobe_write_opcode stores the int3 byte by allocating new page, copying the contents of the old page over and updating it with int3 byte.. then calls __replace_page to put new page in place should that be enough also for 5 bytes update? the cpu executing that exact page will page fault and get the new updated page? I discussed with Oleg and got this understanding, I might be wrong hm what if the cpu is just executing the address in the middle of the uprobe's original instructions and the page gets updated.. I need to check more on this ;-) > > Once patched there will be no simulation of insns or kernel traps. > Just normal user code that calls into trampoline, that calls sys_bpf, > and returns back. I saw this as generic uprobe enhancement, should it be sys_bpf syscall, not a some generic one? we will call all the uprobe's handlers/consumers thanks, jirka
