Hi Peter, Thanks for providing the detailed reproduce. Now I can see the error (after I synced to 6.12.0-rc6, I was using rc5). I'll look into that and report back. > I have tested your provided method, but the AutoFDO profile (lld does not get lto-sample-profile=$pathtoprofile passed) I see. You also turned on ThinLTO, which I didn't, so the profile was only used during compilation, not passed to lld. Thanks, Han On Mon, Nov 4, 2024 at 9:31 AM Peter Jung <ptr1337@xxxxxxxxxxx> wrote: > > Hi Han, > > I have tested your provided method, but the AutoFDO profile (lld does > not get lto-sample-profile=$pathtoprofile passed) nor Clang as compiler > gets used. > Please replace following PKGBUILD and config from linux-mainline with > the provided one in the gist. The patch is also included there. > > https://gist.github.com/ptr1337/c92728bb273f7dbc2817db75eedec9ed > > The main change I am doing here, is passing following to the build array > and replacing "make all": > > make LLVM=1 LLVM_IAS=1 CLANG_AUTOFDO_PROFILE=${srcdir}/perf.afdo all > > When compiling the kernel with makepkg, this results at the packaging to > following issue and can be reliable reproduced. > > Regards, > > Peter > > > On 04.11.24 05:50, Han Shen wrote: > > Hi Peter, thanks for reporting the issue. I am trying to reproduce it > > in the up-to-date archlinux environment. Below is what I have: > > 0. pacman -Syu > > 1. cloned archlinux build files from > > https://aur.archlinux.org/linux-mainline.git the newest mainline > > version is 6.12rc5-1. > > 2. changed the PKGBUILD file to include the patches series > > 3. changed the "config" to turn on clang autofdo > > 4. collected afdo profiles > > 5. MAKEFLAGS="-j48 V=1 LLVM=1 CLANG_AUTOFDO_PROFILE=$(pwd)/perf.afdo" \ > > makepkg -s --skipinteg --skippgp > > 6. install and reboot > > The above steps succeeded. > > You mentioned the error happens at "module_install", can you instruct > > me how to execute the "module_install" step? > > > > Thanks, > > Han > > > > On Sat, Nov 2, 2024 at 12:53 PM Peter Jung<ptr1337@xxxxxxxxxxx> wrote: > >> > >> > >> On 02.11.24 20:46, Peter Jung wrote: > >>> > >>> On 02.11.24 18:51, Rong Xu wrote: > >>>> Add the build support for using Clang's AutoFDO. Building the kernel > >>>> with AutoFDO does not reduce the optimization level from the > >>>> compiler. AutoFDO uses hardware sampling to gather information about > >>>> the frequency of execution of different code paths within a binary. > >>>> This information is then used to guide the compiler's optimization > >>>> decisions, resulting in a more efficient binary. Experiments > >>>> showed that the kernel can improve up to 10% in latency. > >>>> > >>>> The support requires a Clang compiler after LLVM 17. This submission > >>>> is limited to x86 platforms that support PMU features like LBR on > >>>> Intel machines and AMD Zen3 BRS. Support for SPE on ARM 1, > >>>> and BRBE on ARM 1 is part of planned future work. > >>>> > >>>> Here is an example workflow for AutoFDO kernel: > >>>> > >>>> 1) Build the kernel on the host machine with LLVM enabled, for example, > >>>> $ make menuconfig LLVM=1 > >>>> Turn on AutoFDO build config: > >>>> CONFIG_AUTOFDO_CLANG=y > >>>> With a configuration that has LLVM enabled, use the following > >>>> command: > >>>> scripts/config -e AUTOFDO_CLANG > >>>> After getting the config, build with > >>>> $ make LLVM=1 > >>>> > >>>> 2) Install the kernel on the test machine. > >>>> > >>>> 3) Run the load tests. The '-c' option in perf specifies the sample > >>>> event period. We suggest using a suitable prime number, > >>>> like 500009, for this purpose. > >>>> For Intel platforms: > >>>> $ perf record -e BR_INST_RETIRED.NEAR_TAKEN:k -a -N -b -c > >>>> <count> \ > >>>> -o <perf_file> -- <loadtest> > >>>> For AMD platforms: > >>>> The supported system are: Zen3 with BRS, or Zen4 with amd_lbr_v2 > >>>> For Zen3: > >>>> $ cat proc/cpuinfo | grep " brs" > >>>> For Zen4: > >>>> $ cat proc/cpuinfo | grep amd_lbr_v2 > >>>> $ perf record --pfm-events RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k > >>>> -a \ > >>>> -N -b -c <count> -o <perf_file> -- <loadtest> > >>>> > >>>> 4) (Optional) Download the raw perf file to the host machine. > >>>> > >>>> 5) To generate an AutoFDO profile, two offline tools are available: > >>>> create_llvm_prof and llvm_profgen. The create_llvm_prof tool is part > >>>> of the AutoFDO project and can be found on GitHub > >>>> (https://github.com/google/autofdo), version v0.30.1 or later. The > >>>> llvm_profgen tool is included in the LLVM compiler itself. It's > >>>> important to note that the version of llvm_profgen doesn't need to > >>>> match the version of Clang. It needs to be the LLVM 19 release or > >>>> later, or from the LLVM trunk. > >>>> $ llvm-profgen --kernel --binary=<vmlinux> -- > >>>> perfdata=<perf_file> \ > >>>> -o <profile_file> > >>>> or > >>>> $ create_llvm_prof --binary=<vmlinux> --profile=<perf_file> \ > >>>> --format=extbinary --out=<profile_file> > >>>> > >>>> Note that multiple AutoFDO profile files can be merged into one via: > >>>> $ llvm-profdata merge -o <profile_file> <profile_1> ... > >>>> <profile_n> > >>>> > >>>> 6) Rebuild the kernel using the AutoFDO profile file with the same config > >>>> as step 1, (Note CONFIG_AUTOFDO_CLANG needs to be enabled): > >>>> $ make LLVM=1 CLANG_AUTOFDO_PROFILE=<profile_file> > >>>> > >>>> Co-developed-by: Han Shen<shenhan@xxxxxxxxxx> > >>>> Signed-off-by: Han Shen<shenhan@xxxxxxxxxx> > >>>> Signed-off-by: Rong Xu<xur@xxxxxxxxxx> > >>>> Suggested-by: Sriraman Tallam<tmsriram@xxxxxxxxxx> > >>>> Suggested-by: Krzysztof Pszeniczny<kpszeniczny@xxxxxxxxxx> > >>>> Suggested-by: Nick Desaulniers<ndesaulniers@xxxxxxxxxx> > >>>> Suggested-by: Stephane Eranian<eranian@xxxxxxxxxx> > >>>> Tested-by: Yonghong Song<yonghong.song@xxxxxxxxx> > >>>> Tested-by: Yabin Cui<yabinc@xxxxxxxxxx> > >>>> Tested-by: Nathan Chancellor<nathan@xxxxxxxxxx> > >>>> Reviewed-by: Kees Cook<kees@xxxxxxxxxx> > >>> Tested-by: Peter Jung<ptr1337@xxxxxxxxxxx> > >>> > >> The compilations and testing with the "make pacman-pkg" function from > >> the kernel worked fine. > >> > >> One problem I do face: > >> When I apply a AutoFDO profile together with the PKGBUILD [1] from > >> archlinux im running into issues at "module_install" at the packaging. > >> > >> See following log: > >> ``` > >> make[2]: *** [scripts/Makefile.modinst:125: > >> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/arch/x86/kvm/kvm.ko] > >> Error 1 > >> make[2]: *** Deleting file > >> '/tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/arch/x86/kvm/kvm.ko' > >> INSTALL > >> /tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/crypto/cryptd.ko > >> make[2]: *** Waiting for unfinished jobs.... > >> ``` > >> > >> > >> This can be fixed with removed "INSTALL_MOD_STRIP=1" to the passed > >> parameters of module_install. > >> > >> This explicitly only happens, if a profile is passed - otherwise the > >> packaging works without problems. > >> > >> Regards, > >> > >> Peter Jung > >> >
