Note: a resend due to https://lore.kernel.org/dwarves/Z4AWJBNsGJvBU7ZY@x1/ This is v4 of the series aiming to speed up parallel reproducible BTF encoding. This version mostly addresses feedback from Jiri Olsa on v3. A notable adition is a patch 10/10, which changes func_states in btf_encoder from a list to an array. Testing: vmlinux=/home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1 PATH=$(realpath build):$PATH ./tests/tests 1: Validation of BTF encoding of functions; this may take some time: Ok 2: Default BTF on a system without BTF: Ok 3: Flexible arrays accounting: pahole: type 'nft_pipapo_elem' not found pahole: type 'tls_rec' not found pahole: type 'fuse_direntplus' not found pahole: type 'nft_rhash_elem' not found pahole: type 'nft_hash_elem' not found pahole: type 'nft_bitmap_elem' not found pahole: type 'ipt_standard' not found pahole: type 'nft_rule_dp_last' not found pahole: type 'ip6t_standard' not found pahole: type 'ipt_error' not found pahole: type 'ip6t_error' not found pahole: type 'nft_rbtree_elem' not found Ok 4: Check that pfunct can print btf_decl_tags read from BTF: Ok 5: Pretty printing of files using DWARF type information: Ok 6: Parallel reproducible DWARF Loading/Serial BTF encoding: Ok The warnings about not found types are also present at pahole/next, so not related to this patchset. Performance check. This patchset (always reproducible): Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 5,788.22 msec cpu-clock:u # 3.776 CPUs utilized ( +- 0.17% ) 1.53288 +- 0.00334 seconds time elapsed ( +- 0.22% ) pahole/next (d444eb6), parallel non-reproducible: Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 10,462.38 msec cpu-clock:u # 6.678 CPUs utilized ( +- 0.15% ) 1.56670 +- 0.00548 seconds time elapsed ( +- 0.35% ) pahole/next (d444eb6), parallel reproducible: Performance counter stats for '/home/theihor/dev/dwarves/build/pahole -J -j24 --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs,reproducible_build --btf_encode_detached=/dev/null --lang_exclude=rust /home/theihor/kernels/bpf-next/kbuild-output/.tmp_vmlinux1' (13 runs): 6,399.88 msec cpu-clock:u # 3.164 CPUs utilized ( +- 0.22% ) 2.02269 +- 0.00359 seconds time elapsed ( +- 0.18% ) v3: https://lore.kernel.org/dwarves/20241221012245.243845-1-ihor.solodrai@xxxxx/ v2: https://lore.kernel.org/dwarves/20241213223641.564002-1-ihor.solodrai@xxxxx/ v1: https://lore.kernel.org/dwarves/20241128012341.4081072-1-ihor.solodrai@xxxxx/ Alan Maguire (2): btf_encoder: simplify function encoding btf_encoder: separate elf function, saved function representations Ihor Solodrai (8): btf_encoder: free encoder->secinfo in btf_encoder__delete btf_encoder: introduce elf_functions struct type btf_encoder: introduce elf_functions_list btf_encoder: remove skip_encoding_inconsistent_proto dwarf_loader: introduce cu->id dwarf_loader: multithreading with a job/worker model btf_encoder: clean up global encoders list btf_encoder: switch func_states from a list to an array btf_encoder.c | 662 +++++++++++++++++++----------------- btf_encoder.h | 7 +- btf_loader.c | 2 +- ctf_loader.c | 2 +- dwarf_loader.c | 335 ++++++++++++------ dwarves.c | 44 --- dwarves.h | 20 +- pahole.c | 230 ++----------- pdwtags.c | 3 +- pfunct.c | 3 +- tests/reproducible_build.sh | 5 +- 11 files changed, 623 insertions(+), 690 deletions(-) -- 2.47.1
