> On May 6, 2022, at 11:50 PM, Song Liu <songliubraving@xxxxxx> wrote: > > > >> On Apr 27, 2022, at 11:48 PM, Song Liu <songliubraving@xxxxxx> wrote: >> >> Hi Linus, >> >> Thanks for your thorough analysis of the situation, which make a lot of >> sense. >> >>> On Apr 27, 2022, at 6:45 PM, Linus Torvalds <torvalds@xxxxxxxxxxxxxxxxxxxx> wrote: >>> >>> On Wed, Apr 27, 2022 at 3:24 PM Song Liu <songliubraving@xxxxxx> wrote: >>>> >>>> Could you please share your suggestions on this set? Shall we ship it >>>> with 5.18? >>> >>> I'd personally prefer to just not do the prog_pack thing at all, since >>> I don't think it was actually in a "ready to ship" state for this >>> merge window, and the hugepage mapping protection games I'm still >>> leery of. >>> >>> Yes, the hugepage protection things probably do work from what I saw >>> when I looked through them, but that x86 vmalloc hugepage code was >>> really designed for another use (non-refcounted device pages), so the >>> fact that it all actually seems surprisingly ok certainly wasn't >>> because the code was designed to do that new case. >>> >>> Does the prog_pack thing work with small pages? >>> >>> Yes. But that wasn't what it was designed for or its selling point, so >>> it all is a bit suspect to me. >> >> prog_pack on small pages can also reduce the direct map fragmentation. >> This is because libbpf uses tiny BPF programs to probe kernel features. >> Before prog_pack, all these BPF programs can fragment the direct map. >> For example, runqslower (tools/bpf/runqslower/) loads total 7 BPF programs >> (3 actual programs and 4 tiny probe programs). All these programs may >> cause direct map fragmentation. With prog_pack, OTOH, these BPF programs >> would fit in a single page (or even share pages with other tools). > > Here are some performance data from our web service production benchmark, > which is the biggest service in our fleet. We compare 3 kernels: > > nopack: no bpf_prog_pack; IOW, the same behavior as 5.17 > 4kpack: use bpf_prog_pack on 4kB pages (same as 5.18-rc5) > 2mpack: use bpf_prog_pack on 2MB pages > > The benchmark measures system throughput under latency constraints. > 4kpack provides 0.5% to 0.7% more throughput than nopack. > 2mpack provides 0.6% to 0.9% more throughput than nopack. > > So the data has confirmed: > 1. Direct map fragmentation has non-trivial impact on system performance; > 2. While 2MB pages are preferred, bpf_prog_pack on 4kB pages also gives > Significant performance improvements. Please note that 0.5% is a huge improvement for our fleet. I believe this is also significant for other companies with many thousand servers. Thanks, Song
