Thanks for the explanation. On Thu, Nov 7, 2024 at 6:58 AM Masahiro Yamada <masahiroy@xxxxxxxxxx> wrote: > > On Thu, Nov 7, 2024 at 4:00 AM Rong Xu <xur@xxxxxxxxxx> wrote: > > > > On Wed, Nov 6, 2024 at 8:09 AM Masahiro Yamada <masahiroy@xxxxxxxxxx> wrote: > > > > > > On Sun, Nov 3, 2024 at 2:51 AM Rong Xu <xur@xxxxxxxxxx> wrote: > > > > > > > > Hi, > > > > > > > > This patch series is to integrate AutoFDO and Propeller support into > > > > the Linux kernel. AutoFDO is a profile-guided optimization technique > > > > that leverages hardware sampling to enhance binary performance. > > > > Unlike Instrumentation-based FDO (iFDO), AutoFDO offers a user-friendly > > > > and straightforward application process. While iFDO generally yields > > > > superior profile quality and performance, our findings reveal that > > > > AutoFDO achieves remarkable effectiveness, bringing performance close > > > > to iFDO for benchmark applications. > > > > > > > > Propeller is a profile-guided, post-link optimizer that improves > > > > the performance of large-scale applications compiled with LLVM. It > > > > operates by relinking the binary based on an additional round of runtime > > > > profiles, enabling precise optimizations that are not possible at > > > > compile time. Similar to AutoFDO, Propeller too utilizes hardware > > > > sampling to collect profiles and apply post-link optimizations to improve > > > > the benchmark’s performance over and above AutoFDO. > > > > > > > > Our empirical data demonstrates significant performance improvements > > > > with AutoFDO and Propeller, up to 10% on microbenchmarks and up to 5% > > > > on large warehouse-scale benchmarks. This makes a strong case for their > > > > inclusion as supported features in the upstream kernel. > > > > > > > > Background > > > > > > > > A significant fraction of fleet processing cycles (excluding idle time) > > > > from data center workloads are attributable to the kernel. Ware-house > > > > scale workloads maximize performance by optimizing the production kernel > > > > using iFDO (a.k.a instrumented PGO, Profile Guided Optimization). > > > > > > > > iFDO can significantly enhance application performance but its use > > > > within the kernel has raised concerns. AutoFDO is a variant of FDO that > > > > uses the hardware’s Performance Monitoring Unit (PMU) to collect > > > > profiling data. While AutoFDO typically yields smaller performance > > > > gains than iFDO, it presents unique benefits for optimizing kernels. > > > > > > > > AutoFDO eliminates the need for instrumented kernels, allowing a single > > > > optimized kernel to serve both execution and profile collection. It also > > > > minimizes slowdown during profile collection, potentially yielding > > > > higher-fidelity profiling, especially for time-sensitive code, compared > > > > to iFDO. Additionally, AutoFDO profiles can be obtained from production > > > > environments via the hardware’s PMU whereas iFDO profiles require > > > > carefully curated load tests that are representative of real-world > > > > traffic. > > > > > > > > AutoFDO facilitates profile collection across diverse targets. > > > > Preliminary studies indicate significant variation in kernel hot spots > > > > within Google’s infrastructure, suggesting potential performance gains > > > > through target-specific kernel customization. > > > > > > > > Furthermore, other advanced compiler optimization techniques, including > > > > ThinLTO and Propeller can be stacked on top of AutoFDO, similar to iFDO. > > > > ThinLTO achieves better runtime performance through whole-program > > > > analysis and cross module optimizations. The main difference between > > > > traditional LTO and ThinLTO is that the latter is scalable in time and > > > > memory. > > > > > > > > This patch series adds AutoFDO and Propeller support to the kernel. The > > > > actual solution comes in six parts: > > > > > > > > [P 1] Add the build support for using AutoFDO in Clang > > > > > > > > Add the basic support for AutoFDO build and provide the > > > > instructions for using AutoFDO. > > > > > > > > [P 2] Fix objtool for bogus warnings when -ffunction-sections is enabled > > > > > > > > [P 3] Adjust symbol ordering in text output sections > > > > > > > > [P 4] Add markers for text_unlikely and text_hot sections > > > > > > > > [P 5] Enable –ffunction-sections for the AutoFDO build > > > > > > > > [P 6] Enable Machine Function Split (MFS) optimization for AutoFDO > > > > > > > > [P 7] Add Propeller configuration to the kernel build > > > > > > > > Patch 1 provides basic AutoFDO build support. Patches 2 to 6 further > > > > enhance the performance of AutoFDO builds and are functionally dependent > > > > on Patch 1. Patch 7 enables support for Propeller and is dependent on > > > > patch 2 to patch 4. > > > > > > > > Caveats > > > > > > > > AutoFDO is compatible with both GCC and Clang, but the patches in this > > > > series are exclusively applicable to LLVM 17 or newer for AutoFDO and > > > > LLVM 19 or newer for Propeller. For profile conversion, two different > > > > tools could be used, llvm_profgen or create_llvm_prof. llvm_profgen > > > > needs to be the LLVM 19 or newer, or just the LLVM trunk. Alternatively, > > > > create_llvm_prof v0.30.1 or newer can be used instead of llvm-profgen. > > > > > > > > Additionally, the build is only supported on x86 platforms equipped > > > > with PMU capabilities, such as LBR on Intel machines. More > > > > specifically: > > > > * Intel platforms: works on every platform that supports LBR; > > > > we have tested on Skylake. > > > > * AMD platforms: tested on AMD Zen3 with the BRS feature. The kernel > > > > needs to be configured with “CONFIG_PERF_EVENTS_AMD_BRS=y", To > > > > check, use > > > > $ cat /proc/cpuinfo | grep “ brs” > > > > For the AMD Zen4, AMD LBRV2 is supported, but we suspect a bug with > > > > AMD LBRv2 implementation in Genoa which blocks the usage. > > > > > > > > For ARM, we plan to send patches for SPE-based Propeller when > > > > AutoFDO for Arm is ready. > > > > > > > > Experiments and Results > > > > > > > > Experiments were conducted to compare the performance of AutoFDO-optimized > > > > kernel images (version 6.9.x) against default builds.. The evaluation > > > > encompassed both open source microbenchmarks and real-world production > > > > services from Google and Meta. The selected microbenchmarks included Neper, > > > > a network subsystem benchmark, and UnixBench which is a comprehensive suite > > > > for assessing various kernel operations. > > > > > > > > For Neper, AutoFDO optimization resulted in a 6.1% increase in throughput > > > > and a 10.6% reduction in latency. UnixBench saw a 2.2% improvement in its > > > > index score under low system load and a 2.6% improvement under high system > > > > load. > > > > > > > > For further details on the improvements observed in Google and Meta's > > > > production services, please refer to the LLVM discourse post: > > > > https://discourse.llvm.org/t/optimizing-the-linux-kernel-with-autofdo-including-thinlto-and-propeller/79108 > > > > > > > > Thanks, > > > > > > > > Rong Xu and Han Shen > > > > > > > > > I applied this series to linux-kbuild. > > > > > > > Thanks for taking the patch! > > > > > As I mentioned before, I do not like #ifdef because > > > it hides (not fixes) issues only for default cases. > > > > We followed the suggestion and removed most of the #if (or #ifdef) in > > the linker script. > > I just checked: there are two #ifdef remaining: > > (1) in the propeller patch for .llvm_bb_addr_map > > (2) in linker script patch for arch/sparc/kernel/vmlinux.lds.S. > > > > I think it's likely safe to remove the checks for head_64.o in > > non-SPARC64 builds and .llvm_bb_addr_map symbols in non-propeller builds. > > > > SPARC64 builds should always produce head_64.o, and non-SPARC64 > > builds shouldn't. > > > > Propeller builds always generate .llvm_bb_addr_map symbols, and the > > linker will omit the section if it's empty in non-propeller builds. > > > > Keeping the checks is harmless and might slightly reduce linker > > workload for matching. > > But If you'd prefer to remove them, I'm happy to provide a patch. > > > I am talking about the #ifdef in include/asm-generic/vmlinux.lds.h > > > Yeah, it is me who (reluctantly) accepted cb87481ee89d. > > Now, the #ifdef has become a little more complicated. > The default case is safe, but there are hidden issues. > > Some issues are easy to fix, so I sent some patches. > https://lore.kernel.org/linux-kbuild/20241106161445.189399-1-masahiroy@xxxxxxxxxx/T/#t > https://lore.kernel.org/linux-kbuild/20241106161445.189399-1-masahiroy@xxxxxxxxxx/T/#m4e4fa70386696e903b68d3fe1d7277e9a63fbefe > https://lore.kernel.org/linux-kbuild/20241107111519.GA15424@willie-the-truck/T/#mccf6d49ddd11c90dcc583d7a68934bb3311da880 I did notice the issues for .data.* -- that is one of the reasons we separated text from data in our patch. > > For example, see e41f501d3912. > > When CONFIG_LD_DEAD_CODE_DATA_ELIMINATION=y or > CONFIG_LTO_CLANG=y or CONFIG_AUTOFDO_CLANG=y or > CONFIG_PROPELLER_CLANG=y, the .text.startup sections > will go to TEXT_MAIN instead of INIT_TEXT. > This is not a fatal issue, but we cannot reuse memory for .text.startup > sections. > > Removing the #ifdef (i.e. reverting cb87481ee89d) is more difficult > because we need to take a closer look at potential impacts for all > architectures. I'm not sure if there is a naming convention for section names in the kernel. For special sections, we should avoid using .text.* or .data.*, instead, using "..', or use other prefixes. The compiler can generate sections names like .text.hot.*", ".text.unknown.*", ".text.unlikely.*", ".text.split.*", ".text.startup." or ".text.exit. It seems we've addressed most of them except .text.startup and .text.exit. For text.startup and .text.exit, have you considered renaming the sections within the linker script -- they are fixed strings and should be able to be renamed. > > I understood you did not want to take a risk to break random architectures, > so I decided to postpone the #ifdef issue and accept your patch set. Thanks for the understanding! > > -- > Best Regards > Masahiro Yamada
