== Summary ==
All binaries (executables and shared libraries) are annotated with an ELF note that identifies the rpm distributing this file.
== Owner ==
* Name: [[User:Zbyszek|Zbigniew Jędrzejewski-Szmek]]
* Email: zbyszek@xxxxxxxxx
* Name: Lennart Poettering
* Email: mzsrqben@xxxxxxxxxxxx
== Detailed Description ==
See [https://github.com/systemd/systemd/issues/18433 systemd issue #18433] for discussion and implementation proposals.
Programs crash. And when they do, they dump core, and we want to tell the user which package, including the version, caused the failure. ELF note `.note.package` will be added to specify package nevra. By embedding the this information directly in the binary object, package nevra is immediately available from a core dump.
=== Existing system: `.note.gnu.build-id` ===
We already have build-ids: every ELF object has a `.note.gnu.build-id` note, and given a core file, we can read the build-id and look it up in the rpm database (`dnf repoquery --whatprovides debuginfo(build-id) = …`) to map it to a package name.
Build-ids are unique and compact and very generic and work as expected in general. But they have some downsides:
* build-ids are not very informative for users. Before the build-id is converted back to the appropriate package, it's completely opaque.
* build-ids require a working rpm database or an internet connection to map to the package name.
Three important cases:
* minimal containers: the rpm database is not installed in the containers. The information about build-ids needs to be stored externally, so package name information is not available immediately, but only after offline processing. The new note doesn't depend on the rpm db in any way.
* handling of a core from a container, where the container and host have different distros
* self-built and external packages: unless a lot of care is taken to keep access to the debuginfo packages, this information may be lost. The new note is available even if the repository metadata gets lost. Users can easily provide equivalent information in a format that makes sense in their own environment. It should work even when rpms and debs and other formats are mixed, e.g. during container image creation.
=== New system: `.note.package` ===
The new note is created and propagated similarly to `.note.gnu.build-id`. The difference is that we inject the information about package nevra from the build system.
The implementation is very simple: `%{build_ldflags}` are extended with a command to insert a custom note as a separate section in an ELF object. See [https://github.com/systemd/package-notes/blob/main/hello.spec hello.spec] for an example. This is done in the default macros, so all packages that use the prescribed link flags will be affected.
The note is a compat json string. This allows the format to be trivially extensible (new fields can be added at will), easy to process (json is extremely popular and parsers are widely available). Using a single field is more space-efficient. With multiple fields the padding and alignment requirements cause unnecessary overhead.
The system was designed with cross-distro collaboration and is flexible enough to identify binaries from different packaging formats and build systems (rpms, debs, custom binaries).
The overhead is about 200 bytes for each ELF object.
If we do this only for executables, then for the whole distro, 5000 × 200 = 1 MB.
If we do it for shared libraries, then the cost will be maybe 4 times higher.
Precise measurements TBD once we know the final implementation and figure out the right repoquery magic.
=== Examples ===
<pre>
$ objdump -s -j .note.package build/libhello.so
build/libhello.so: file format elf64-x86-64
Contents of section .note.package:
02ec 04000000 63000000 7e1afeca 46444f00 ....c...~...FDO.
02fc 7b227479 7065223a 2272706d 222c226e {"type":"rpm","n
030c 616d6522 3a226865 6c6c6f22 2c227665 ame":"hello","ve
031c 7273696f 6e223a22 302d312e 66633335 rsion":"0-1.fc35
032c 2e783836 5f363422 2c226f73 43706522 .x86_64","osCpe"
033c 3a226370 653a2f6f 3a666564 6f726170 :"cpe:/o:fedorap
034c 726f6a65 63743a66 65646f72 613a3333 roject:fedora:33
035c 227d0000 "}..
</pre>
<pre>
$ readelf --notes build/hello | grep "description data" | sed -e "s/\s*description data: //g" -e "s/ //g" | xxd -p -r | jq
readelf: build/hello: Warning: Gap in build notes detected from 0x1091 to 0x10de
readelf: build/hello: Warning: Gap in build notes detected from 0x1091 to 0x10af
readelf: build/hello: Warning: Gap in build notes detected from 0x1091 to 0x119f
{
"type": "rpm",
"name": "hello",
"version": "0-1.fc35.x86_64",
"osCpe": "cpe:/o:fedoraproject:fedora:33"
}
</pre>
<pre>
$ coredumpctl info
PID: 44522 (fsverity)
...
Package: fsverity-utils/1.3-1
build-id: ac89bf7175b04d7eec7f6544a923f45be111f0be
Message: Process 44522 (fsverity) of user 1000 dumped core.
Found module /home/bluca/git/fsverity-utils/libfsverity.so.0 with build-id: fa40fdfb79aea84167c98ca8a89add9ac4f51069
Metadata for module /home/bluca/git/fsverity-utils/libfsverity.so.0 owned by FDO found: {
"packageType" : "deb",
"package" : "fsverity-utils",
"packageVersion" : "1.3-1"
}
Found module linux-vdso.so.1 with build-id: aba08e06103f725e26f1d7c178fb6b76a564a35d
Found module libpthread.so.0 with build-id: e91114987a0147bd050addbd591eb8994b29f4b3
Found module libdl.so.2 with build-id: d3583c742dd47aaa860c5ae0c0c5bdbcd2d54f61
Found module ld-linux-x86-64.so.2 with build-id: f25dfd7b95be4ba386fd71080accae8c0732b711
Found module libcrypto.so.1.1 with build-id: 749142d5ee728a76e7cdc61fd79d2311a77405a2
Found module libc.so.6 with build-id: 18b9a9a8c523e5cfe5b5d946d605d09242f09798
Found module fsverity with build-id: ac89bf7175b04d7eec7f6544a923f45be111f0be
Metadata for module fsverity owned by FDO found: {
"packageType" : "deb",
"package" : "fsverity-utils",
"packageVersion" : "1.3-1"
}
Stack trace of thread 44522:
#0 0x00007fe7c8af26f4 __GI___nanosleep (libc.so.6 + 0xc66f4)
#1 0x00007fe7c8af262a __sleep (libc.so.6 + 0xc662a)
#2 0x00005608481407dd main (fsverity + 0x27dd)
#3 0x00007fe7c8a5009b __libc_start_main (libc.so.6 + 0x2409b)
#4 0x000056084814094a _start (fsverity + 0x294a)
</pre>
== Benefit to Fedora ==
A simple and reliable way to gather information about package versions of crashing programs is added. It enhances, instead of replacing, any existing mechanisms. It gives us a nicer story for debugging programs in containers.
== Scope ==
* Proposal owners:
** create a specification (Initial version DONE: [https://systemd.io/COREDUMP_PACKAGE_METADATA COREDUMP_PACKAGE_METADATA])
** write a script to generate the package note (Initial version DONE: [https://github.com/systemd/package-notes/blob/main/generate-package-notes.py generate-package-notes.py])
** provide a patch for `redhat-rpm-config` to insert appropriate compilation options
** extend systemd's coredumpctl to extract and display this information (DONE: [https://github.com/systemd/systemd/pull/19135 PR #19135])
** submit pull request to Packaging Guidelines
* Other developers:
** possibly add support in abrt?
* Release engineering: There should be no impact.
* Policies and guidelines:
The new flags should be mentioned in Packaging Guidelines.
* Trademark approval: N/A (not needed for this Change)
N/A
* Alignment with Objectives:
It might be relevant for Minimization. Even though it increases the image size a tiny bit, it makes minimized images work a bit better.
== Upgrade/compatibility impact ==
No impact.
== How To Test ==
<pre>
$ bash -c 'kill -SEGV $$'
$ coredumpctl
TIME PID UID GID SIG COREFILE EXE SIZE PACKAGE
Mon 2021-03-01 14:37:22 CET 855151 1000 1000 SIGSEGV present /usr/bin/bash 51.7K bash-5.1.0-2.fc34.x86_64
</pre>
== User Experience ==
`coredumpctl` should display information about package versions.
== Dependencies ==
None.
== Contingency Plan ==
* Contingency mechanism: Remove the new compilation flags. Rebuild any packages that were build with the new flags.
* Contingency deadline: Beta freeze.
* Blocks release? No.
== Documentation ==
* https://systemd.io/COREDUMP_PACKAGE_METADATA/
* https://github.com/systemd/package-notes
See also [[Changes/DebuginfodByDefault]].
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
