On 9/28/20 8:50 AM, Jan Kratochvil wrote:
On Mon, 28 Sep 2020 12:31:59 +0200, Mark Wielaard wrote:
If you want to make -fdebug-types-sections the default you really
should work with the upstream GCC developers to figure out why they
don't want that.
I haven't seen that, according to Richard Biener from GCC
-fdebug-types-section is a normally supported GCC feature:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=88878#c6
I am aware only of Jakub Jelinek who is against -fdebug-types-section.
I should also state he is the author of DWZ.
-fdebug-types-section a supported option in the sense that it's in the
compiler and we'll fix bugs in it when we can. But the GCC community
doesn't really test that option and it's known to be broken with LTO.
With the move to LTO for F33 we'd need to fix whatever is broken with
-fdebug-types-section (and I don't know precisely what that is) before
we could even consider turning it on.
Jakub has a ton of experience with debug info generation in GCC as
well as with the dwarf standardization process and based on those
experiences he thinks dwz is a technically better solution. You
disagree. I would strongly advise against hinting that Jakub is biased
against -fdebug-types-section simply because he is the author of dwz.
That just makes you look combative.
If GCC is unable to support such a trivial feature as -fdebug-types-section
then Fedora should really already switch to the standard compiler. It will
come sooner or later anyway. This deviation from standard tools just causes
continuous troubles such as:
[fesco] Issue #2020: Firefox is switching from gcc to clang/llvm
https://pagure.io/fesco/issue/2020#comment-671672
I don't think it argues for that *at all*. Not even close.
I do think we want to bring GCC and LLVM to parity and remove the Fedora
policy which favors GCC. I've still got a TODO to write the actual
changes for the Fedora packaging guidelines to make that happen.
Trying to override upstream defaults in Fedora without understanding why
upstream decided on the current defaults isn't a good idea IMHO.
You know very well Fedora already overrides upstream GCC defaults a lot:
-O2 -g -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection
And, not surprisingly, our team has had significant input on the options
we're using *and* the GCC implementation of those options. We make
recommendations based on our experiences. That same experience would
lead us to recommending against -fdebug-types-section at this time.
And DWZ is even a project unrelated to GCC so calling for upstream defaults
would really call for dropping DWZ.
Again, I don't see that at all. dwz is a separate and distinct step in
the build process. It's not overriding anything in the compiler at all.
I totally get that it is frustrating if you worked for a long time on a
new feature to support some DWARF constructs for lldb
It is in no way a feature as it does not bring any user visible improvement.
It is only a compatibility with marginally (0.1%) used file format with
disputable benefits.
and your aren't able to get the patches in shape to be accepted upstream.
That is a repeated lie I have never even suggested. LLDB is fine accepting my
DWZ patches. I understand you are used to difficulty of upstreaming patches in
GNU projects but that is not the case of LLVM. In fact LLDB is a completely
different world in accepting patches than GDB where it was taking me even 10+
years to get some patches accepted. For GDB you need to learn first how to do
the ancient ineffective and bug-prone coding practices, force yourself to
really execute them to become a global maintainer and only then you manage to
get patches checked in.
You are repeatedly trying to make it look as the problem is upstreaming DWZ
support. That is not any problem. The problem is the DWZ itself as it just
isn't worth the effort of supporting it in all the consumers.
But -fdebug-types-section isn't a viable alternative right now IMHO.
As I am repeating again and again I just find DWZ too complicated for both
production and consumption for so little gain (size reduction) it achieves.
So before I complicate the LLDB codebase by the DWZ support and make it
a catch-up game for Red Hat, Fedora and others forever (as Apple+Google is
never going to use DWZ and they know why) I am trying by this Change to save
a lot of time for everyone.
The years of engineering time I have already spent on DWZ and the years of
engineering time I will have to spend on its future maintenance and reworks
(for clang DWARF) could be better spent on improving the debugging experience.
We are no longer living in 80es where few saved bytes of size were critical
whether the debugger will be able to run or not. Apparently GNU developers
still haven't realized that change.
dwz fixed a real problem for real customers. If you're going to suggest
replacing it, you have to propose something that continues to solve
those problems and hopefully brings them additional benefits. I
realize that's a RHEL issue and that Fedora may ultimately want to do
something different. But implying that the problem isn't real is just
silly.
In fact, if you look at some packages in Fedora, their debuginfo is so
large that they actually turn off debug symbols or throttle back the
level of debugging information they generate so that they can
successfully link.
But I am
happy people now know about your patches and seem to find them useful.
"useful" means with my patches they can workaround the Fedora problem of
encumbering its debuginfo by DWZ.
And this question is not about the existing patches. That waste of engineering
time has been already done. It is about the future waste of time maintaining
compatibility with the DWZ format almost nobody (0.1%) cares about.
This is bordering on flame-bait.
You say it is difficult to support DWARF partial units as generated by
dwz in lldb, but dwz doesn't really do anything non-standard (and GCC
with LTO also generates partial units).
"standard" means that the DWZ specification has been accepted into DWARF-5
standard. IMO that was a mistake. It may have happened because the DWZ author
is a member of the DWARF standard committee. Apparently nobody has so far
implemented any reasonable/effective consumer for DWARF-5 DWZ otherwise they
would put some restrictions into the standard.
[ ... ]
I'm happy to see some concrete suggestions for fixing dwz. But I find
the overall tone of this message fairly combative. I hope that's not
your intention.
To make DWZ better consumable it needs to have the partial units separately
parseable. That way they can be shared at IR level and not just at DWARF level
That means:
* DW_TAG_partial_unit should have DW_AT_language.
* DW_TAG_partial_unit must contain only types (struct/class).
Currently they contain for example also static constant variables but when
you parse such independent DW_TAG_partial_unit into which dictionary you
will register such variable? That makes no sense.
Currently DWZ has benefits only with DWZ common file. Without DWZ common files
DWZ produces 1.6% files bigger than -fdebug-types-section. Therefore if the
DWZ common files saving of 3.3% per Fedora distribution size is worth it then
the existing DWZ tool should be dropped anyway as one can use normal
-fdebug-types-section and one can just write a simple tool moving DW_UT_type
units to the DWZ common file and converting DW_AT_signature declarations to
DW_FORM_ref_sup4 and we are done.
It would certainly be good to improve the on-disk distro size. 99% of
the time nobody cares about debugging system bits. But we can't take a
major regression in the debuginfo size to achieve that and
-fdebug-types-section isn't functional for Fedora. So the only paths
forward I see are to either fix -fdebug-types-section or improve dwz.
DWARF standard sometimes makes mistakes, for example .debug_pubnames and
.debug_pubtypes were never really usable and DWARF-5 removed them. It may be
perfectly possible the DWZ extension of DWARF-5 will be removed in DWARF-6 or
future DWARF standards as it turns out it is not worth the format complexity.
That some text has been accepted into the DWARF standard does not mean much.
Sure. We all make mistakes and standards bodies are no different. If
you feel strongly that it was a mistake, then get involved in the
process and try to correct it.
It is all about engineering effort. I agree if the support of DWZ was trivial
(or there were unlimited engineering resources) then DWZ is really better than
-fdebug-types-section (except it would need a DWZ tool with less bugs and
better coding practices). But reality shows the DWZ support is not trivial
engineering resources for Fedora are very limited and so we have to decide
whether the serious effort to support DWZ is better spent on DWZ or on making
the debuggers better/really usable.
Given how much you propose DWZ apparently the 3.3% Fedora distribution size
increase (if DWZ is dropped) is considered as too much. Obviously if the size
increase was just let's say 0.1% it would be acceptable to drop DWZ - do you
agree? So where is your size limit where the years-effort of supporting DWZ is
worth it? I would say my limit would be maybe 20%, far above the 3.3%.
Putting my Red Hat hat on, I get pushback from PM on *any* size
increases in the RHEL space. While I often question the technical
reasons behind why our customers are pushing for marginal (IMO)
improvements, reality is they care. As much as we'd like to be in a
world were a 1% increase in distro size doesn't matter, that's not the
actual world we live in.
And our RHEL customers absolutey do care about the size of debuginfo
becuause it affects link times.
Anyway, I'll put my Fedora hat back on :-)
For example during Fedora Package Review Process do some packages get rejected
because they would make the distribution too large? Not worth of including
such new package? I am not aware of such decision and it even sounds funny to
me. But that is what you choose here by enforcing DWZ no matter how little
savings it has.
I'm not involved in the new package review process. But I think you're
mixing apples and potatoes here. You're talking about a change which
will increase the size of every package. While a new package typically
only shows up for people that actually install it. The two really
aren't comparable.
DWZ is a nice engineering idea and a nice engineering challenge. But it has no
meaning for end-users. This is why I have switched from GDB to LLVM as
Google&Apple are solving real end-user problems and not artificial engineering
challenges just to have some nice coding time. But in the end I end up stuck
on another GNU non-sense (this time DWZ) needing to be supported by LLVM in
Fedora only for compatibility reasons.
If your complaint is that partial unit DIEs are missing some for your use
case essential attributes,
No, my complaint is that DWZ is just not worth it.
Your opinion. Right now mine is that it is very much worth it. That may
change one day, but right now dwz is the best solution we've got.
I do find your statistics per package useful because they show dwz is
in general effective by producing at least 20% (more) on-disk size
reduction, even though there are some packages where dwz doesn't seem
as effective as it could be. We definitely should investigate those
issues.
And does the 20% reduction of installed size when whole *-debuginfo.rpms get
installed is really worth the delay of 3.5 years of DWARF-5, delay of 3.5
years of LLDB index (.debug_names), years of incompatible LLVM and years of
wasted engineering time?
I do not think so. Maybe Fedora/FESCo thinks differently and this is what I am
asking by my Change about.
I can't speak for FESCO.
But I don't really understand why you then focus on the zstd compressed
rpms (even if even those favor dwz).
I do not see how it favors dwz, I haven't seen and I haven't done
a measurement of separate *.debug files compression of DWZ
vs. -fdebug-types-section. My guess is there will not be a big difference for
DWZ vs. -fdebug-types-section size ratio.
Or why for the debuginfod use case you seem to do the opposite, not take
into account that the http debuginfod server will compress the files before
sending over the network.
See the paragraph above.
-fdebug-types-section has better compression ratio than DWZ for
*-debuginfo.rpm because for *.rpm the compression is applied for all its files
together. The compression algorithm then finds the same parts of separate
*.debug files similar to what DWZ does.
ACK. But again until -fdebug-types-section works with LTO, it's a
non-starter IMHO.
Finally I am interested in your proposal to implement a different way
to reduce the size of DIE trees by eliminating "unused" DIEs. It is
hard to predict what effect that would have without seeing an
implementation (in theory GCC with LTO would not actually generate
debuginfo for unused functions).
It is fastest to wait a few days for the LLVM presentation:
https://whova.com/embedded/session/llvm_202010/1193947/
2) Fragmenting the DWARF to Enable Dead Debug Data Elimination
Killing dead DIEs is good and doing it as early in the overall pipeline
is better.
But I think that can be done separate
from your proposal and combined with other size reduction techniques.
I agree with Mark here. IT's an independent improvement.
By that LLVM dead DIES reduction talk I wanted to just show apparent nobody
cares too much about few percents of the *-debuginfo.rpm size - otherwise it
would be already coded/used. Or if there wasn't DWZ Fedora could already
switch to DWARF-5 which saves probably more size than DWZ does.
I wasn't even aware we had dead dies. THat doesn't mean I don't care,
it just means I wasn't aware. Others may be aware, but have higher
priority items on their TODO lists. Stating that "nobody cares too much
..." isn't helpful.
Jeff
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
