On Fri, May 19, 2023 at 5:07 PM Iñaki Ucar <iucar@xxxxxxxxxxxxxxxxx> wrote: > > On Fri, 19 May 2023 at 16:55, Dan Horák <dan@xxxxxxxx> wrote: > > > > On Fri, 19 May 2023 16:27:23 +0200 > > Iñaki Ucar <iucar@xxxxxxxxxxxxxxxxx> wrote: > > > > > Hi, > > > > > > Do we know why some ppc64le builds take so much? And with "so much" I > > > mean 7-10x the time for a "normal" run. Examples: 2 hours for [1] vs. > > > 20 hours for [2]. > > > > > > And if we do know the cause, is there any way to predict it in order > > > to avoid the %check section? > > > > > > [1] https://koji.fedoraproject.org/koji/taskinfo?taskID=98395536 > > > [2] https://koji.fedoraproject.org/koji/taskinfo?taskID=98395502 > > > > seems the build got restarted, perhaps due OOM on the builder, and > > actual build time was 12h, perhaps the builder or the vmhost were > > overloaded. Do you see the long build times in recent builds too? Both > > examples are from March. > > Here's one: https://koji.fedoraproject.org/koji/taskinfo?taskID=101326951 > Compared to: https://koji.fedoraproject.org/koji/taskinfo?taskID=101327166 > > And I suspect this is the cause of the large number of random errors > we've been experiencing with R packages recently. I think that the R > check command has a timeout that is triggered when a ppc64le build > takes too much (specifically when rebuilding package vignettes). > Things seem to have gone back to normal as I noticed them and disabled > vignette rebuilds in most (all?) of them. But these random extreme > delays are annoying, especially in packages with heavy tests. I've been experiencing similar issues with ppc64le koji builds for the past few weeks. They are now by far the slowest architecture, and sometimes the build tasks are seemingly just "hanging" or "stuck", often for half an hour or longer. Most frequently the tasks look locked up doing disk IO, for example, during dnf's or rpm's transaction checks (i.e. when installing the buildroot or build dependencies). I've seen tasks frequently get stuck at "dnf: Running transaction check" for *ages* (i.e. 30 minutes or longer), and after the builds on all other architectures were long done, they *sometimes* un-stuck themselves after a while and the build progressed (albeit very very slowly). At other times, the builds were just stuck completely - in these cases I've asked releng to free the ppc64le build to restart it, and that solved the problem (most of the time) ... Asking on the fedora-infra IRC / Matrix channel, nirik mentioned that it might be caused by recent kernels (6.1 or 6.2), with 6.3 looking better at first glance. Fabio _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
