F40 Change Proposal: DNFConditionalFilelists (System-Wide)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



= DNF: Do not download filelists by default =

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

Wiki: https://fedoraproject.org/wiki/Changes/DNFConditionalFilelists


== Summary ==
Change the DNF behavior to not download filelists by default. These
metadata, which describe all the files contained within each package,
are unnecessary in the majority of use cases. Additionally, these
metadata files can be large in size, leading to a significant slowdown
in the user experience.

== Owner ==
* Name: [[User:jkolarik| Jan Kolarik]]
* Email: jkolarik@xxxxxxxxxx



== Detailed Description ==
Until now, filelists were always downloaded together with other
metadata. This was hardcoded and unable to change from the outside of
DNF.

With these changes, we are proposing to not download the filelists
metadata by default. This default behavior can be modified through the
new DNF configuration option. Additionally, specific commands can
override this behavior and request loading the filelists metadata at
runtime using the existing demands object in DNF.

Note that after this change, users can still use DNF without filelists
metadata when querying file provides located in `/usr/bin`,
`/usr/sbin` or `/etc` directories.

== Feedback ==


== Benefit to Fedora ==
As DNF is integral to various infrastructure tasks like package
building and installation, testing environment creation, and server
integration tests, this change significantly reduces processing time
and resource usage for these processes.

This change reduces the RAM requirements of the DNF process,
addressing existing issues when running the Fedora system on
low-memory machines such as the Raspberry Pi (see f.e.
[https://bugzilla.redhat.com/show_bug.cgi?id=1907030 Bug 1907030]).

Also, omitting the filelists metadata download overall decreases the
costs of a Fedora mirror server operation.

== Scope ==
* Proposal owners:
** libdnf
*** Modify the `Repo` object to enable conditional filelists metadata download
*** Introduce a new main configuration option to set the default behavior
** dnf
*** Enable configuration of filelists download from commandline, DNF
commands and DNF plugins
*** Implement filename pattern argument detection heuristics

* Other developers: <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
** Dependencies using the existing DNF C interface may need to adapt
if they expect the filelists metadata to be available and explicitly
request loading filelists using the existing API due to this change:
*** PackageKit
*** microdnf
*** API users

* Release engineering: N/A

* Policies and guidelines:
** Package maintainers must follow Fedora's packaging guidelines,
particularly concerning file dependency specifications (see
[https://docs.fedoraproject.org/en-US/packaging-guidelines/#_file_and_directory_dependencies
here])

* Trademark approval: N/A


* Alignment with Community Initiatives: N/A (no currently active initiatives)


== Upgrade/compatibility impact ==
In general, applying these changes should not affect any existing user
workflows and no additional manual changes are required. However, the
absence of filelists might create an issue with packages that are not
correctly packaged or originate from third-party repositories. In the
current Fedora release repository, there are only a few such packages,
see the [https://bugzilla.redhat.com/show_bug.cgi?id=2180842#c8
comment] in [https://bugzilla.redhat.com/show_bug.cgi?id=2180842 Bug
2180842].

== How To Test ==
When using DNF commands without a filename pattern passed as the
argument, filelists metadata should not be downloaded from the remote
repositories and should not be needed for the command execution. This
can be tested with the following steps:
* Clean the local metadata cache (`dnf clean metadata`)
* Run a DNF command not involving the filename spec (e.g. `dnf repoquery rpm`)
* Verify that no `*-filelists.*` metadata files were downloaded inside
the cache subdirectories (by default under the `/var/cache/dnf` for
root)
* Check the command works as expected
The same should also apply to RPM package arguments (files ending with
`.rpm` extension).

When using DNF commands with a filename pattern passed as the
argument, filelists metadata should be downloaded from the remote
repositores as before.

== User Experience ==
Large filelists could be over 200MB in size. It could take 1-2 minutes
to download which is greatly slowing down the user experience.

For many operations the filelists metadata are not needed, so
downloading them is wasting the resources. Without filelists being
downloaded, DNF performance will be improved significantly, mainly
regarding the network, CPU and disk space resources. Metadata download
size will be reduced by about 60%. The improvement includes
deployments of customer built RPMS to containers that have no need for
filelists level dependencies.


== Dependencies ==
No changes should be required for any package depending on DNF to
implement this behavior.

== Contingency Plan ==
* Contingency mechanism: Change the configuration option to download
the filelists by default
* Contingency deadline: Branch Fedora Linux 40 from Rawhide
* Blocks release? No

== Documentation ==
Links to the relevant DNF CLI and API documentation sections will be
provided here once the related pull request is created.

== Release Notes ==



-- 
Aoife Moloney

Fedora Operations Architect

Fedora Project

Matrix: @amoloney:fedora.im

IRC: amoloney
_______________________________________________
devel-announce mailing list -- devel-announce@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-announce-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue




[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Index of Archives]     [Fedora Announce]     [Fedora Users]     [Fedora Kernel]     [Fedora Testing]     [Fedora Formulas]     [Fedora PHP Devel]     [Kernel Development]     [Fedora Legacy]     [Fedora Maintainers]     [Fedora Desktop]     [PAM]     [Red Hat Development]     [Gimp]     [Yosemite News]

  Powered by Linux