Wiki - https://fedoraproject.org/wiki/Changes/ReproduciblePackageBuilds Discussion.fpo - https://discussion.fedoraproject.org/t/f41-change-proposal-reproducible-package-builds-system-wide/112740 == Summary == A post-build cleanup is integrated into the RPM build process so that common causes of build irreproducibility in packages are removed, making most of Fedora packages reproducible. == Owner == * Name: Davide Cavalca * Name: Neil Hanlon * Name: [[User:churchyard|Miro Hrončok]] * Name: [[User:zbyszek|Zbigniew Jędrzejewski-Szmek]] * Email: dcavalca@xxxxxxxxxxxxxxxxx * Email: neil at shrug.pw * Email: mhroncok at redhat.com * Email: zbyszek at in.waw.pl == Detailed Description == As of 2023 there is an active effort to implement [https://docs.fedoraproject.org/en-US/reproducible-builds/ Reproducible builds] in Fedora. Reproducible builds will allow our users to be able to independently verify that the RPMs have not been tampered with (either maliciously or via hardware/software fault): someone can do an independent rebuild of a package and confirm that they get identical binaries when building with the same versions of the compiler and other tools. This Change allows us to move forward in this direction by removing the common sources of irreproducibility. [https://github.com/keszybz/add-determinism add-determinism] is a Rust program which, as its name suggests, adds determinism to files that are given as input by attempting to standardize metadata contained in binary or source files to ensure consistency and clamping to $SOURCE_DATE_EPOCH in all instances. `add-determinism` is the "Fedora version" of [https://salsa.debian.org/reproducible-builds/strip-nondeterminism strip-nondeterminism] from the Debian project. Since strip-nondeterminism is written in perl, it is undesirable for use in Fedora, as we don't want to pull perl in the buildroot for every package. It's worth noting that this Change does not intend to impose any specific reproducibility requirements on Fedora packages. Once this Change is implemented and we have been through a mass rebuild and can verify that the common causes of irreproducibility have indeed been removed, we can consider further steps. But that will be at least one release later. This change does add a small amount of time to the processing of RPMs at the end of a build. Accordingly, packages containing large quantities or sizes of files can be slower, but this effect is not expected to be noticeable. `add-determinism` takes steps to ensure it does not interfere with other buildroot post processors like `mangle-shebangs`, `python-hardlink`, `python-bytecompile`. It defaults to not doing any modifications in case it doesn't understand the input file or there are any other problems. A mechanism to opt-out will be provided: to either completely disable the postprocessing step or to disable specific "handlers" (i.e. implementations of cleanup for specific file types, for example static archives). See [https://github.com/keszybz/add-determinism/blob/main/rpm/macros.build-reproducibility macros.build-reproducibility]. === Related Changes === * [https://fedoraproject.org/wiki/Changes/ReproducibleBuildsClampMtimes Clamp build mtimes to SOURCE_DATE_EPOCH] * [https://fedoraproject.org/wiki/Changes/RPM-4.20 RPM 4.20] — this pulls in changes to `%autosetup -S git` which removed a source of irreproducibility. == Feedback == == Benefit to Fedora == Adding determinism (i.e., removing non-determinsim) enables the Fedora community to have confidence that, if given the same source code, build environment, build instructions, and metadata from the build artifacts, any party can recreate copies of the artifacts that are identical except for the signatures and some parts of metadata. Reproducibility of builds leads to packages of higher quality. It turns out that quite often those irreproducible bits are caused by an error or sloppiness in the code. In particular, any dependence on architecture in noarch packages is almost always unwanted and/or a bug. Test builds that check reproducibility will expose such instances. Reproducibility of builds makes it easier to develop packages: when a small change is made and a package is rebuilt (in the same environment), then with a reproducible package, the only difference is directly caused by the change. If the package is different every time it is rebuilt, making a comparison is much harder. Build reproducibility for noarch ''sub''packages solves the problem where package builds on different architectures are different, causing mock to reject the whole build. In particular, this issue occurs for [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/#_byte_compilation_reproducibility pyc files]. This will now be solved without requiring opt-in from individual packages. == Scope == * Proposal Owners: ** Integrate `add-determinism` as a BuildRoot Policy script ** Add a dependency on `marshalparser` to `python3` (probably conditionalized on `rpm-build`) * Other Developers: ** Test their packages with the additional phase, report problems ** Potentially integrate changes to packages to enable reproducibility * Release Engineering: Ideally we want this to happen before the mass rebuild, but that is not strictly required. * Policies and Guidelines: Fedora Packaging Guidelines should be updated to include information on the add-determinism BuildRoot Policy. User documentation should be amended to include instructions on how to verify reproducibility for a given package, and what packages are known to be non-reproducible, and how to opt-out. * Trademark approval: N/A (not needed for this Change) * Alignment with Community Initiatives: All software and requests are consistent with the decision process and similar across other groups in Fedora. The Fedora Reproducibility Working group began at Flock 2023 in Cork. == Upgrade/compatibility impact == No impact is expected. == How To Test == To test on the level of individual files: * install `add-determinism` * call `SOURCE_DATE_EPOCH=… add-determinism -v ./path/to/file` To test package builds: * build a local copy of `redhat-rpm-config` with https://src.fedoraproject.org/rpms/redhat-rpm-config/pull-request/293 * install `add-determinism` * build packages ;) (This can be done on a normal system or in a mock chroot.) == User Experience == No impact is expected. == Dependencies == == Contingency Plan == * Contingency mechanism: ** In case of major problems, disable the change in `redhat-rpm-config`. ** In case of problems with specific packages, opt-out by setting a macro. * Contingency deadline: No limit really. * Blocks release? No. == Documentation == * [https://docs.fedoraproject.org/en-US/reproducible-builds/ Fedora Reproducible Builds] * [https://github.com/keszybz/add-determinism/blob/main/rpm/macros.build-reproducibility add-determinism macros.build-reproducibility] * [https://github.com/keszybz/add-determinism/tree/main?tab=readme-ov-file#build-postprocessor-to-reset-metadata-fields-for-build-reproducibility add-determinism README] == Release Notes == Fedora package builds are now more deterministic, bringing the distribution closer to the goal of achieving fully reproducible builds for all of its packages. -- Aoife Moloney Fedora Operations Architect Fedora Project Matrix: @amoloney:fedora.im IRC: amoloney -- _______________________________________________ devel-announce mailing list -- devel-announce@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-announce-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue -- _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
