F40 Change Proposal: Drop Delta RPMs (System-Wide)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Wiki https://fedoraproject.org/wiki/Changes/Drop_Delta_RPMs

== Summary ==

Stop producing Delta RPMs during the compose process, and disable deltarpm
support in the default configuration of DNF / DNF5.

== Owner ==

* Name: [[User:decathorpe| Fabio Valentini]]
* Email: decathorpe@xxxxxxxxx


* Targeted release:
[https://docs.fedoraproject.org/en-US/releases/f40/ Fedora Linux 40]
* Last updated: {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}
* [<will be assigned by the Wrangler> devel thread]
* FESCo issue: <will be assigned by the Wrangler>
* Tracker bug: <will be assigned by the Wrangler>
* Release notes tracker: <will be assigned by the Wrangler>


== Detailed Description ==

Delta RPMs are an optimization that reduces the amount of data that needs to be
downloaded for package updates, by only downloading the parts of the package
that have changed between the locally installed version and the new version, and
reassembling a complete RPM package for the new version locally.

Due to the way they are generated during the compose process, it is not possible
to produce Delta RPMs for all packages that are involved in an upgrade (since
only one previous version is considered for delta generation). This can lead to
upgrades that involve hundreds of packages, but only a tiny fraction of them (or
none at all) also have appropriate Delta RPMs available in the repository.

Additionally, reconstruction of the new version of the RPM from the currently
installed version and the Delta RPM can fail, causing an additional download of
the complete RPM for the new version, wasting bandwidth.

On top of these issues, the presence of Delta RPMs in package repositories also
inflates the size of the repository metadata, which needs to be downloaded by
all users, whether the actual upgrade involves Delta RPMs or not. This most
affects users that upgrade via GUI utilities like GNOME Software - because the
PackageKit libdnf backend has no support for Delta RPMs at all, or users of
OSTree based variants / spins of Fedora - where Delta RPMs are not used either.

According to early feedback from Release Engineering, it looks like it will not
be feasible to address the shortcomings of Delta RPMs as they are currently
implemented, and since they often no longer result in a net reduction in
download sizes for upgrades, this Change proposes both to disable the generation
of DeltaRPMs during the compose process, and to disable the deltarpm support in
dnf / dnf5 by default.

=== Some statistics ===

I have collected data for system upgrades with dnf across three different Fedora
installs (Fedora 38 Workstation, Fedora 37 KDE, and Fedora 38 KDE, respectively,
all with the "updates-testing" repository enabled) for two months, with varying
intervals between upgrades (between one day and one month) in an attempt to
capture different situations. To summarize the results:

* Delta RPMs were available for only 25 out of 42 upgrades
* Delta RPMs saved 7.5 MB / 8% of downloads on average
* Delta RPMs saved 22.5 MB of download on average (if there were no failures)
* Delta RPMs wasted 52.7 MB of download on average (if there were failures)

As an anecdote, most of the savings are attributable to one package (firefox).

For reference, the current sizes of repository metadata for Fedora 38 (as of
2023-09-07):

* fedora: 86.8 MB
* updates: 33.2 MB
* updates-testing: 13.3 MB

Note that refreshing repository metadata only causes deltas to be downloaded,
not a full re-download (due to using zchunk for repodata).

== Feedback ==

N/A

== Benefit to Fedora ==

Benefits for Fedora infrastucture:

* simplification of the compose process for "updates" and
"updates-testing" repositories (generation of Delta RPMs will be
skipped)
* reduction of storage requirements in Fedora infrastructure and on
repository mirrors (both due to smaller metadata and dropped Delta
RPMs)
* reduction in bandwith use for repository metadata updates

Benefits for Fedora users:

* more reliable upgrades (i.e. failures to re-assemble RPMs from Delta
RPMs are eliminated)
* reduction in bandwith use for repository metadata updates
* reduction in CPU usage for package upgrades (local re-assembly of
RPMs from on-disk data and Delta RPMs is eliminated)

== Scope ==

* Proposal owners:
** modify compose configuration to skip generation of Delta RPMs for Fedora 40+
** modify default configuration for dnf / dnf5 to disable deltarpm support
* Other developers:
** coordination with dnf and dnf5 developers and / or package maintainers
* Release engineering: [https://pagure.io/releng/issue/11665 releng#11665]
** merge compose changes to skip generation of Delta RPMs for Fedora 40+
* Policies and guidelines: N/A (not needed for this Change)
* Trademark approval: N/A (not needed for this Change)
* Alignment with Community Initiatives: N/A (no currently active initiatives)

== Upgrade/compatibility impact ==

Installations that get upgraded to Fedora 40 should not be negatively affected
by this Change. The configuration change for dnf / dnf5 will only be
automatically applied if there were no modifications to
<code>/etc/dnf/dnf.conf</code>. However, since the repositories will no longer
contain Delta RPMs, this configuration change is not strictly required.

Alternatively, the default dnf / dnf5 configuration could be changed at the code
level instead of changes to configuration files.

== How To Test ==

Users should be able to verify that support for Delta RPMs is disabled in dnf
on Fedora 40, for example by checking for <code>deltarpm = 0</code> in the
output of <code>dnf config-manager --dump | grep deltarpm</code>.

== User Experience ==

Users who upgrade their systems with GNOME Software (or other similar tools)
should not be affected by this change, other than faster and smaller repository
metadata downloads.

Users who upgrade via dnf / dnf5 should see a similar improvement. Additionally,
there will be no more wasted downloads due to failed reassembly of RPMs from
Delta RPMs.

== Dependencies ==

* Release Engineering changes (modifications of the compose process)
* dnf / dnf5 package changes (disabling deltarpm support, either via
configuration changes, or changing the default from <code>true</code>
to <code>false</code> in the code itself)

== Contingency Plan ==

* Contingency mechanism:

If the compose configuration cannot be modified to skip generation of Delta RPMs
in time for Fedora 40, changing the default configuration in dnf / dnf5 will
still cause some of the benefits outlined above (except for those caused by the
smaller repository metadata sizes).

If the default dnf configuration cannot be changed in time for Fedora 40, the
benefits outlined above should still apply in almost all circumstances.

If neither changes can be implemented in time, the change can be postponed to
the next Fedora release without causing problems.

* Contingency deadline: Fedora 40 Beta Freeze

* Blocks release? No

== Documentation ==

N/A

== Release Notes ==

As of Fedora 40, package repositories no longer contain Delta RPMs, and package
managers (dnf, dnf5) have disabled support for deltarpms by default. They are
not useful in many common circumstances (i.e. upgrades via GNOME Software or
on OSTree based variants), and often no longer even cause less data to be
downloaded for upgrades.




-- 
Aoife Moloney

Product Owner

Community Platform Engineering Team

Red Hat EMEA

Communications House

Cork Road

Waterford
_______________________________________________
devel-announce mailing list -- devel-announce@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-announce-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux