On Fri, 25 Sep 2020 01:35:43 +0200, Mark Wielaard wrote:
> Replying since I am mentioned by name in this proposal and it seems to
> argue for removing a feature I am currently working on to make sure it
> works correctly with GCC11 if it switches to producing DWARF5 by
> default.
The problem is you are not working on DWARF-5 features produced by LLVM so
even your planned DWZ is still not usable for LLVM-built binaries.
In the case DWZ is used for GCC-built binaries what to do with LLVM-built
binaries?
(1) Build them without DWZ nor -fdebug-types-section.
They will get up to 2x as big.
(2) Build them with -fdebug-types-section.
Then the distro tools need to be -fdebug-types-section compatible anyway
(which they already are modulo testing).
(3) You were proposing to build them as DWARF-4.
That regresses functionality of Fedora compared to other OSes.
It is also unfair to restrict LLVM due to a deficiency of DWZ.
> On Thu, Sep 24, 2020 at 11:59:44AM -0400, Ben Cotton wrote:
> As others pointed out dwz is widely used. It is used by almost every
> distro in some form and even freedesktop.org flatpaks use dwz for
> their debuginfo.
As Debian really uses DWZ (as pointed out by Florian Weimer) I have updated
the Change text now.
> The upstream project is actively maintained.
Fedora has been waiting for DWARF-5 for 3.5 years due to DWZ. So I do not find
it as maintained. But I have removed the sentence if there is a disagreement
on it.
> Even though there are just 3 committers (including me) the project is still
> seeing ~2.5 commits a week (about 130 in a year).
Still existing bugs are not fixed for years as nobody looks at build.log's.
> > There exist several methods
> > how to make the *-debuginfo.rpms at least a bit smaller. Fedora 18
> > started using DWZ tool (from [[Features/DwarfCompressor]]) while
> > [https://gcc.gnu.org/pipermail/gcc-patches/2008-August/246281.html
> > Google implemented] the same goal in a different way called
> > -fdebug-types-section.
>
> Note that these methods are not in conflict. Both started out as GNU
> extensions but have been standardized since.
They are in conflict as DWZ still does not support -fdebug-types-section
(after 12 years of -fdebug-types-section and 8 years of DWZ).
In binaries built with -fdebug-types-section DWZ cannot use -m DWZ common file
and in such case DWZ produces 1.6% bigger Fedora distro *-debuginfo.rpm than
simple -fdebug-types-section. That means running DWZ after
-fdebug-types-section is pointless.
> > so its support is missing in tools like
> > [https://lldb.llvm.org/ LLDB],
>
> But you have been maintaining an out of tree patch for several years
> to support partial units and supplemental files (both of which dwz
> produces and are now standard DWARF). It would be good if you
> upstreamed those patches.
That's the question why I submitted this Change. From my point of view the DWZ
technology makes no sense and so I find a better fix to drop DWZ.
I am interesting in opinion of Fedora / FESCo.
> > or binutils readelf.
>
> binutils readelf is used as the main tool in the dwz testsuite. It
> certainly supports both partial units and supplemental files. Also
> note the the -wK (=follow-links) option [it isn't on by default, maybe
> it should] that resolves alt links.
I was not aware of it (it is from 2017-11-15). I have updated the Change text.
Yes, it definitely should be on by default.
> I don't know of any other tool which doesn't support either partial
> units or supplemental files, since both are (now) standard DWARF.
GCC also does not support most of DWARF-5 after it is standardized for 3.5
years (only these days you started implementing better DWARF-5 support).
And the DWARF-5 standard even still does not specify .debug_names for DWZ,
that has been apparently forgotten.
> > * DWZ disadvantage: DWZ requires 8x times more complicated (LoC count)
> > support in consumers than -fdebug-types-section.
>
> I have worked on support for both in various consumers and didn't find
> one method more difficult to support than the other. Maybe debug-types
> was actually more difficult. Because the representation changed from
> separate section in DWARF4, to intermingled with other .debug_info
> units in DWARF5. Making supporting objects that contained mixed
> versions a bit of a pain.
It is easy for primitive tools like GDB which do DWARF->IR (Internal
Representation) conversion for whole CUs (Compilation Units = per .o file) and
not per DIE (one element like a variable/type/function). That was fine in 80s
but not now with big C++ template libraries. That means on real C++ code GDB
consumes about 20GB of memory and 5 minutes of runtime to print a variable.
That does not happen much for Fedora packages as most of the packages are not
in C++ and the C++ ones are not much heavily templatized (there are
exceptions). LLDB makes many indexes of the DIEs to be effective and with DWZ
all of those need to be modified to track also DWZ parent importer units
indicating dictionaries where the IRs should be placed.
> > * DWZ disadvantage: DWZ cannot update LLVM .debug_names index which
> > can be generated only by clang (it cannot be regenerated later for
> > DWZ-compressed file)
>
> dwz does support .gdb_index, the pre-standardized DWARF5 .debug_names
> variant. gdb hasn't switched to supporting .debug_names yet and
> .gdb_index does work with DWARF5. I don't think the gdb maintainers
> would mind you adding .debug_names support if you believe it to be
> better than .gdb_index.
GDB does not need .debug_names because it does the simple but ineffective full
CU IR conversion. .gdb_index cannot be used by effective LLDB as .gdb_index
does not contain the essential DIE offsets at all.
> It is a not very flexible design with a somewhat high overhead (somewhat
> reduced in DWARF5 by not making it part of a separate section).
Whether the section is in ".debug_info" or ".debug_types" is a few lines of
code. But tracking importing CUs are thousands of lines of code.
I understand DWZ is not a problem for trivial utilities, that is not the case
of LLDB.
> If you think it really is a good idea to use them more broadly and by
> default then you should probably start by getting consensus from the
> upstream gcc developers that it should be enabled by default.
I could discuss making -fdebug-types-section the default for clang as I have
switched to clang in 2013 and I have never looked back.
> I don't think it is in conflict with also using dwz. But it will probably
> require some extra work.
The whole DWZ question is whether to use DWZ common files. DWZ common files
are in /usr/lib/debug/.dwz/ , there is always one file per *-debuginfo.rpm and
they do save some *.debug size by placing big class definitions to a single
place for the whole rpm. This way DWZ has 6.35% (for DWARF-4) smaller
*-debuginfo.rpm size (although the 6.35% has stddev +/-11%, it depends A LOT
on which package you DWZ).
OTOH when downloading *.debug files separately (such as by debuginfod
https://sourceware.org/elfutils/Debuginfod.html
) the DWZ common files make the total download sizes bigger by 6% in average
compared to -fdebug-types-section.
So the DWZ common files have some advantages and disadvantages. And given
I believe nobody cares about few percents of *-debuginfo.rpm size I do not
find it worth the trouble.
In the case one no longer uses DWZ common files (dwz option -m) then DWZ
produces *-debuginfo.rpm 1.6% bigger than more simple -fdebug-types-section.
So without DWZ common files I find obvious DWZ would really make no sense.
Thanks for reply,
Jan
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
