On Monday, October 14th, 2024 at 3:31 AM, Alan Maguire <alan.maguire@xxxxxxxxxx> wrote:
>
>
> This reduces peak memory utilization during BTF vmlinux encoding,
> replacing ELF section-sized malloc()s with mmap()s.
>
> For -j1 peak memory resident set size drops from
>
> Maximum resident set size (kbytes): 1069220
>
> ...to:
>
> Maximum resident set size (kbytes): 701888
>
> ...with this patch.
>
> For -j8, peak memory resident set size drops from
>
> Maximum resident set size (kbytes): 1121172
>
> ...to:
>
> Maximum resident set size (kbytes): 963516
>
> Signed-off-by: Alan Maguire alan.maguire@xxxxxxxxxx
>
> ---
> btf_encoder.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/btf_encoder.c b/btf_encoder.c
> index 5954238..9599faf 100644
> --- a/btf_encoder.c
> +++ b/btf_encoder.c
> @@ -1879,7 +1879,7 @@ static int btf_encoder__tag_kfuncs(struct btf_encoder *encoder)
> goto out;
> }
>
> - elf = elf_begin(fd, ELF_C_READ, NULL);
> + elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
> if (elf == NULL) {
> elf_error("Cannot update ELF file");
> goto out;
> --
> 2.43.5
Alan, I can confirm this reduces max resident set size when BTF
encoding vmlinux. Great improvement from a single flag change! See
below.
for j in 1 2 4 8 16 32 64; do
echo -n '-j$j: '
/usr/bin/time -v \
./pahole -J -j$j \
--btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs \
--btf_encode_detached=/dev/null \
--lang_exclude=rust \
~/repo/bpf-dev-docker/linux/.tmp_vmlinux1 \
2>&1 | grep 'Maximum resident set size'
done
Current next (91bcd1d):
-j1: Maximum resident set size (kbytes): 1600036
-j2: Maximum resident set size (kbytes): 1623380
-j4: Maximum resident set size (kbytes): 1635824
-j8: Maximum resident set size (kbytes): 1631412
-j16: Maximum resident set size (kbytes): 1711976
-j32: Maximum resident set size (kbytes): 1799856
-j64: Maximum resident set size (kbytes): 1976184
This patch on top of next (91bcd1d):
-j1: Maximum resident set size (kbytes): 640344
-j2: Maximum resident set size (kbytes): 695272
-j4: Maximum resident set size (kbytes): 769772
-j8: Maximum resident set size (kbytes): 895728
-j16: Maximum resident set size (kbytes): 984908
-j32: Maximum resident set size (kbytes): 1148420
-j64: Maximum resident set size (kbytes): 1573944
This patch on top of elf_functions patch (with segfault fixes) [1]:
-j1: Maximum resident set size (kbytes): 642992
-j2: Maximum resident set size (kbytes): 695432
-j4: Maximum resident set size (kbytes): 746888
-j8: Maximum resident set size (kbytes): 851208
-j16: Maximum resident set size (kbytes): 884232
-j32: Maximum resident set size (kbytes): 930632
-j64: Maximum resident set size (kbytes): 1127976
Tested-by: Ihor Solodrai <ihor.solodrai@xxxxx>
[1]: https://lore.kernel.org/dwarves/20241009233520.989453-1-ihor.solodrai@xxxxx/
