On 3/30/21 12:43 PM, Nico Kadel-Garcia wrote:
On Tue, Mar 30, 2021 at 4:46 AM Panu Matilainen <pmatilai@xxxxxxxxxx> wrote:
On 3/29/21 8:17 PM, Michel Alexandre Salim wrote:
Hi again,
On Fri, 2021-03-26 at 17:29 -0700, Michel Alexandre Salim wrote:
[snip]
Right now I'm just overriding _smp_build_ncpus to 1, but there is a
more elegant solution I'd like to propose:
What if one can declaratively set the required RAM per build job --
either with a single macro, or maybe two if the LTO usecase requires
even more RAM. e.g. to declare each core might take up to 8 GB:
%global _smp_build_ram_per_cpu 8192
then in case this is run on our aarch64 builder with 40GB RAM,
dynamically take the minimum of the existing _smp_build_ncpus (which
AIUI is determined by the number of cores on the machine) and (amount
of RAM / _smp_build_ram_per_cpu), in this case capping the actual
number passed to -j to 5.
Is there interest in having this be available? I could imagine it
might
be useful for other resource-intensive package builds e.g. for
Chromium.
Thanks to Dennis, Dan and Miroslav for the helpful pointers!
Short-term I'll look into adopting something similar to the Ceph
script, but medium-term - maybe we can get the scriptlet into redhat-
rpm-config and then eventually have it in RPM itself once Panu's patch
is reworked?
(Having it in redhat-rpm-config or some other RPM is probably needed
anyway, for older supported releases, so ideally we use the same macros
and only override them in redhat-rpm-config if they were undefined).
cc:ing Panu for context on the RPM PR status.
There's no progress on that front, other than occasionally thinking
about it.
It might not be a bad idea to be able to put an actual BuildRequire on
the amount of memory (and other similar - disk space also comes to mind)
into specs.
Let's *not*. The amount of space saved in storage by putting socks and
underwear in their own separate drawers os often overwhelmed by the
space wasted in giving those drawers sides and a way to slide in and
out. In other words, it creates unnecessary overhead in managing build
environment resources. It's also difficult to predict in advance,
especially for the worst offenders, such as builds whose upstream
components are often pulled in dynamically: "pip install", "cpan",
"mvn", "gradle", "rubygem", and my recent favorite for unpredictable
and only erratically evailble dependencies, "go". While well designed
SRPMs and .spec files to rely only on other RPMs for their
dependencies, many in the field do not. And even when the dependencies
are available, even a small change in "test" procedures can massively
expand RAM and disk use during compilation.
In other words, the memory requirements are likely to diverge,
*massively* and unpredictably. It's safer and simpler to simply
allocate memory generously for build environments.
The point is not that everybody should add such requirements to their
packages, but to have that *option* for those monster packages out there.
- Panu -
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
