Am 13.07.22 um 16:30 schrieb John Reiser:
On 7/11/22 Marius Schwarz wrote:
I have just create(d/ not finished yet, started 15 minutes ago) a
~2.5 GB rpm and found, that rpmbuild is an extrem bottleneck.
IMHO, this is caused by a fileread function which reads files in 32k
blocks, which is very slow and extrem IO intensive. The result is a
task running at 1 core at 100% perma. With changes to larger chunks,
we can speed up so many build tasks on the farm.
Multicore use would also be helpful i.e. while packing the files.
Any counter-arguments ?
If you give the complete package name and URL of the repo,
then more persons may be likely to help investigate.
Specifying a reproducible example is always good.
All issues solved for far. Just to give you (all) an impression, here
are source and result in my test repo:
3,2G /usr/share/pva/vosk-model-de-0.21
[vosk-model-de-0.21]# du -sh *
100M am
12K conf
685M graph
8,2M ivector
4,0K README
2,1G rescore
281M rnnlm
[rescore]# ll
insgesamt 2171812
-rw-r--r-- 1 root root 2115929988 14. Sep 2021 G.carpa <---------
-rw-r--r-- 1 root root 107992138 14. Sep 2021 G.fst
So compressing this 2+ GB file (and others) was slowing down the process
because of the one core compression default.
Building this now takes just ~4-5 minutes on 8 cores and a system doing
other things in parallel.
Resulting in a 1.7 GB rpm :
-rw-r--r-- 1 root root 1758210157 14. Jul 09:44
/home/linux-am-dienstagde/repo/x86_64/fedora/35/pva-vosk-model-de-large-1-2.x86_64.rpm
Luckily, not all vosk language models are not changing frequently and
are not that big, but some are.
If this ever makes it into Fedora repo, it will take a lot of space and
bind resources on builds ;)
@BCotton:
No idea, if you remember, but when i said it will waste 100gb + updates,
in the last year, there were only a few updates to the languages models,
reducing the expected needed space over time a lot.
Best regards,
Marius
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
