Hi Rong,
Glad that you were able to reproduce the issue!
Thanks for finding the root cause as well as the part of the code. This
really helps.
I was able to do a successful packaging with binutils 2.42.
Lets forward this to the binutils tracker and hope this will be soon
solved. :)
I have tested this also on the latest commit
(e1e4078ac59740a79cd709d61872abe15aba0087) and the issue is also
reproducible there.
Thanks for your time! I dont see this as blocker. :)
It gets time to get this series merged :P
Best regards,
Peter
On 05.11.24 08:25, Rong Xu wrote:
We debugged this issue and we found the failure seems to only happen
with strip (version 2.43) in binutil.
For a profile-use compilation, either with -fprofile-use (PGO or
iFDO), or -fprofile-sample-use (AutoFDO),
an ELF section of .llvm.call-graph-profile is created for the object.
For some reasons (like to save space?),
the relocations in this section are of type "rel', rather the more
common "rela" type.
In this case,
$ readelf -r kvm.ko |grep llvm.call-graph-profile
Relocation section '.rel.llvm.call-graph-profile' at offset 0xf62a00
contains 4 entries:
strip (v2.43.0) has difficulty handling the relocations in
.rel.llvm.call-graph-profile -- it silently failed with --strip-debug.
But strip (v.2.42) has no issue with kvm.ko. The strip in llvm (i.e.
llvm-strip) also passes with kvm.ko
I compared binutil/strip source code for version v2.43.0 and v2.42.
The different is around here:
In v2.42 of bfd/elfcode.h
1618 if ((entsize == sizeof (Elf_External_Rela)
1619 && ebd->elf_info_to_howto != NULL)
1620 || ebd->elf_info_to_howto_rel == NULL)
1621 res = ebd->elf_info_to_howto (abfd, relent, &rela);
1622 else
1623 res = ebd->elf_info_to_howto_rel (abfd, relent, &rela);
In v2.43.0 of bfd/elfcode.h
1618 if (entsize == sizeof (Elf_External_Rela)
1619 && ebd->elf_info_to_howto != NULL)
1620 res = ebd->elf_info_to_howto (abfd, relent, &rela);
1621 else if (ebd->elf_info_to_howto_rel != NULL)
1622 res = ebd->elf_info_to_howto_rel (abfd, relent, &rela);
In the 2.43 strip, line 1618 is false and line 1621 is also false.
"res" is returned as false and the program exits with -1.
While in 2.42, line 1620 is true and we get "res" from line 1621 and
program functions correctly.
I'm not familiar with binutil code base and don't know the reason for
removing line 1620.
I can file a bug for binutil for people to further investigate this.
It seems to me that this issue should not be a blocker for our patch.
Regards,
-Rong
On Mon, Nov 4, 2024 at 12:24 PM Han Shen<shenhan@xxxxxxxxxx> wrote:
Hi Peter,
Thanks for providing the detailed reproduce.
Now I can see the error (after I synced to 6.12.0-rc6, I was using rc5).
I'll look into that and report back.
I have tested your provided method, but the AutoFDO profile (lld does
not get lto-sample-profile=$pathtoprofile passed)
I see. You also turned on ThinLTO, which I didn't, so the profile was
only used during compilation, not passed to lld.
Thanks,
Han
On Mon, Nov 4, 2024 at 9:31 AM Peter Jung<ptr1337@xxxxxxxxxxx> wrote:
Hi Han,
I have tested your provided method, but the AutoFDO profile (lld does
not get lto-sample-profile=$pathtoprofile passed) nor Clang as compiler
gets used.
Please replace following PKGBUILD and config from linux-mainline with
the provided one in the gist. The patch is also included there.
https://gist.github.com/ptr1337/c92728bb273f7dbc2817db75eedec9ed
The main change I am doing here, is passing following to the build array
and replacing "make all":
make LLVM=1 LLVM_IAS=1 CLANG_AUTOFDO_PROFILE=${srcdir}/perf.afdo all
When compiling the kernel with makepkg, this results at the packaging to
following issue and can be reliable reproduced.
Regards,
Peter
On 04.11.24 05:50, Han Shen wrote:
Hi Peter, thanks for reporting the issue. I am trying to reproduce it
in the up-to-date archlinux environment. Below is what I have:
0. pacman -Syu
1. cloned archlinux build files from
https://aur.archlinux.org/linux-mainline.git the newest mainline
version is 6.12rc5-1.
2. changed the PKGBUILD file to include the patches series
3. changed the "config" to turn on clang autofdo
4. collected afdo profiles
5. MAKEFLAGS="-j48 V=1 LLVM=1 CLANG_AUTOFDO_PROFILE=$(pwd)/perf.afdo" \
makepkg -s --skipinteg --skippgp
6. install and reboot
The above steps succeeded.
You mentioned the error happens at "module_install", can you instruct
me how to execute the "module_install" step?
Thanks,
Han
On Sat, Nov 2, 2024 at 12:53 PM Peter Jung<ptr1337@xxxxxxxxxxx> wrote:
On 02.11.24 20:46, Peter Jung wrote:
On 02.11.24 18:51, Rong Xu wrote:
Add the build support for using Clang's AutoFDO. Building the kernel
with AutoFDO does not reduce the optimization level from the
compiler. AutoFDO uses hardware sampling to gather information about
the frequency of execution of different code paths within a binary.
This information is then used to guide the compiler's optimization
decisions, resulting in a more efficient binary. Experiments
showed that the kernel can improve up to 10% in latency.
The support requires a Clang compiler after LLVM 17. This submission
is limited to x86 platforms that support PMU features like LBR on
Intel machines and AMD Zen3 BRS. Support for SPE on ARM 1,
and BRBE on ARM 1 is part of planned future work.
Here is an example workflow for AutoFDO kernel:
1) Build the kernel on the host machine with LLVM enabled, for example,
$ make menuconfig LLVM=1
Turn on AutoFDO build config:
CONFIG_AUTOFDO_CLANG=y
With a configuration that has LLVM enabled, use the following
command:
scripts/config -e AUTOFDO_CLANG
After getting the config, build with
$ make LLVM=1
2) Install the kernel on the test machine.
3) Run the load tests. The '-c' option in perf specifies the sample
event period. We suggest using a suitable prime number,
like 500009, for this purpose.
For Intel platforms:
$ perf record -e BR_INST_RETIRED.NEAR_TAKEN:k -a -N -b -c
<count> \
-o <perf_file> -- <loadtest>
For AMD platforms:
The supported system are: Zen3 with BRS, or Zen4 with amd_lbr_v2
For Zen3:
$ cat proc/cpuinfo | grep " brs"
For Zen4:
$ cat proc/cpuinfo | grep amd_lbr_v2
$ perf record --pfm-events RETIRED_TAKEN_BRANCH_INSTRUCTIONS:k
-a \
-N -b -c <count> -o <perf_file> -- <loadtest>
4) (Optional) Download the raw perf file to the host machine.
5) To generate an AutoFDO profile, two offline tools are available:
create_llvm_prof and llvm_profgen. The create_llvm_prof tool is part
of the AutoFDO project and can be found on GitHub
(https://github.com/google/autofdo), version v0.30.1 or later. The
llvm_profgen tool is included in the LLVM compiler itself. It's
important to note that the version of llvm_profgen doesn't need to
match the version of Clang. It needs to be the LLVM 19 release or
later, or from the LLVM trunk.
$ llvm-profgen --kernel --binary=<vmlinux> --
perfdata=<perf_file> \
-o <profile_file>
or
$ create_llvm_prof --binary=<vmlinux> --profile=<perf_file> \
--format=extbinary --out=<profile_file>
Note that multiple AutoFDO profile files can be merged into one via:
$ llvm-profdata merge -o <profile_file> <profile_1> ...
<profile_n>
6) Rebuild the kernel using the AutoFDO profile file with the same config
as step 1, (Note CONFIG_AUTOFDO_CLANG needs to be enabled):
$ make LLVM=1 CLANG_AUTOFDO_PROFILE=<profile_file>
Co-developed-by: Han Shen<shenhan@xxxxxxxxxx>
Signed-off-by: Han Shen<shenhan@xxxxxxxxxx>
Signed-off-by: Rong Xu<xur@xxxxxxxxxx>
Suggested-by: Sriraman Tallam<tmsriram@xxxxxxxxxx>
Suggested-by: Krzysztof Pszeniczny<kpszeniczny@xxxxxxxxxx>
Suggested-by: Nick Desaulniers<ndesaulniers@xxxxxxxxxx>
Suggested-by: Stephane Eranian<eranian@xxxxxxxxxx>
Tested-by: Yonghong Song<yonghong.song@xxxxxxxxx>
Tested-by: Yabin Cui<yabinc@xxxxxxxxxx>
Tested-by: Nathan Chancellor<nathan@xxxxxxxxxx>
Reviewed-by: Kees Cook<kees@xxxxxxxxxx>
Tested-by: Peter Jung<ptr1337@xxxxxxxxxxx>
The compilations and testing with the "make pacman-pkg" function from
the kernel worked fine.
One problem I do face:
When I apply a AutoFDO profile together with the PKGBUILD [1] from
archlinux im running into issues at "module_install" at the packaging.
See following log:
```
make[2]: *** [scripts/Makefile.modinst:125:
/tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/arch/x86/kvm/kvm.ko]
Error 1
make[2]: *** Deleting file
'/tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/arch/x86/kvm/kvm.ko'
INSTALL
/tmp/makepkg/linux-cachyos-rc-autofdo/pkg/linux-cachyos-rc-autofdo/usr/lib/modules/6.12.0-rc5-5-cachyos-rc-autofdo/kernel/crypto/cryptd.ko
make[2]: *** Waiting for unfinished jobs....
```
This can be fixed with removed "INSTALL_MOD_STRIP=1" to the passed
parameters of module_install.
This explicitly only happens, if a profile is passed - otherwise the
packaging works without problems.
Regards,
Peter Jung
