On Thu, Nov 5, 2020 at 12:29 PM Arnaldo Carvalho de Melo
<acme@xxxxxxxxxx> wrote:
>
> Em Thu, Nov 05, 2020 at 11:10:14AM -0800, Andrii Nakryiko escreveu:
> > On Thu, Nov 5, 2020 at 3:42 AM Arnaldo Carvalho de Melo <acme@xxxxxxxxxx> wrote:
> > > Em Wed, Nov 04, 2020 at 08:39:36PM -0800, Andrii Nakryiko escreveu:
> > > > @@ -679,11 +681,11 @@ static int btf_elf__write(const char *filename, struct btf *btf)
> > > > {
> > > > GElf_Shdr shdr_mem, *shdr;
> > > > GElf_Ehdr ehdr_mem, *ehdr;
> > > > - Elf_Data *btf_elf = NULL;
> > > > + Elf_Data *btf_data = NULL;
>
> > > Can you please split this into two patches, one doing just the rename
> > > of btf_elf to btf_data and then moving to btf__new_empty_split()? Eases
> > > reviewing.
>
> > sure, will do in the next version
>
> Thanks!
>
> > > With this split btf code would it be possible to paralelize the encoding
> > > of the modules BTF? I have to check the other patches and how this gets
> > > used in the kernel build process... :-)
>
> > Yes, each module's BTF is generated completely independently. See some
> > numbers in [0].
> >
> > [0] https://patchwork.kernel.org/project/netdevbpf/patch/20201105045140.2589346-4-andrii@xxxxxxxxxx/
>
> I saw it, very good. I wonder if we could manage to also paralelize the
> processing of DWARF compile units in the BTF encoder, like start
> processing and at the end just figure out how many types were in a CU,
> get the highest type id and bump it to + the number of types in the
> current CU, adjust the types, continue, something like that.
A big chunk is just DWARF loading. If it's possible to parallelize
that, it would be a big improvement. To parallelize BTF encoding
itself, it could be possible to generate each individual CU's types in
a separate BTF object, and then merge them together. We'd probably
need a few new APIs (like btf__append_btf() which would add all types
and strings, renumbering everything on the fly). BTF dedup itself is
unlikely to be parallelizable easily, I definitely wouldn't dare ;)
But it's also not a big portion of overall BTF encoding process. DWARF
parsing and string lookup/addition where by far biggest CPU hogs.
>
> - Arnaldo
