On 4/21/21 5:57 AM, Jinpu Wang wrote: > On Wed, Apr 21, 2021 at 1:55 PM Pavel Begunkov <asml.silence@xxxxxxxxx> wrote: >> >> On 4/21/21 12:50 PM, Jinpu Wang wrote: >> [snip] >>>>> Hi Jens, >>>>> The problem with using blktrace at production may cause a performance >>>>> drop ~30%. while with the block stats here, we only see ~3% when >>>>> enabled. >>>> >>>> It's probably was asked before, but let's refresh as the discussion >>>> erupted again. >>>> >>>> I get your problem with blktrace(8), IIRC it definitely can deteriorate >>>> performance if run constantly, but did you try to write a bpf program >>>> that does smarter accumulation in the kernel? Like making bpf to collect >>>> a latency table (right as in your patches do) and flushing it to the >>>> disk periodically? >>> Hi Pavel, >>> >>> Thanks for the suggestion. >>> >>> We did test with ebpf with kprobe in the past (~kernel 4.4/4.14), we >>> saw 10% performance drop, that's the reason we develop this >>> stats patches. >>> >>> But I just did another test with bpftrace on k 5.10.30, I do not see >>> performance lost. >>> It must be ebpf is improving very much since then. >>> >>> So to summarize, we can use bpftrace to do the drop in latest kernel, >>> there is no need to have it build into the kernel. >> >> Perfect, and I'm sure it will be even more convenient for you, for >> instance to gather other stats or do it somehow differently > > Yeah, agree. > Thanks again! Perfect, thanks! I'll drop the series. -- Jens Axboe
