On Mon, Jan 25, 2021 at 01:59:59PM -0800, Kevin Fenzi wrote: > On Mon, Jan 25, 2021 at 12:19:13PM +0000, Peter Robinson wrote: > > On Sun, Jan 24, 2021 at 8:55 PM Kevin Fenzi <kevin@xxxxxxxxx> wrote: > > > > > > On Sun, Jan 24, 2021 at 08:21:48PM +0000, Peter Robinson wrote: > > > > On Sun, Jan 24, 2021 at 7:54 PM Kevin Fenzi <kevin@xxxxxxxxx> wrote: > > > > > > > > > > On Sun, Jan 24, 2021 at 08:38:44PM +0100, Jakub Jelinek wrote: > > > > > > On Sat, Jan 23, 2021 at 03:21:14PM -0800, Kevin Fenzi wrote: > > > > > > > > I've been poking at it more today... > > > > > > > > > > > > > > > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=60286478 > > > > > > > > > Total time 13:19:20 > > > > > > > > > Task time 1:11:53 > > > > > > > > > > > > > > > > Yeah, basically kojid get OOM killed, which causes it to restart and > > > > > > > > koji sees the build never finished and restarts it. ;( > > > > > > > > > > > > > > > > Right now I am going to take them all down and: > > > > > > > > > > > > > > > > * remove the qemu highmem=off (this is not needed in the lpae case for > > > > > > > > us, just non lpae, and might be causing some, but not all issues) > > > > > > > > > > > > > > > > * reduce their memory to 16GB. At 16GB I was able to get a python3.8 > > > > > > > > build cleanly with tests in 40min. Right now with 24GB on the builders > > > > > > > > they OOM kojid (as you see with gcc above). > > > > > > > > > > > > > > > > Hopefully those two things will make them more stable... > > > > > > > > > > > > > > ok. Done. All of them are up with 16gb and no highmem=off. > > > > > > > > > > > > > > I really hope this will stablize them before the mass rebuild. > > > > > > > > > > > > Total time 34:23:40 > > > > > > Task time 1:16:34 > > > > > > > > > > > > Already 5 times OOM killed. > > > > > > > > > > Yep. Been trying other things on it today in staging to get a stable > > > > > builder. :( > > > > > > > > > > I'm now trying (in staging) moving them back to direct boot from uefi. > > > > > If that doesn't work on f33, I will just punt and move them back to f32 > > > > > (like they were before) with a really old kernel. ;( > > > > > > > > maybe try the 5.6.x kernel on the uefi/f33 userspace, are any of the > > > > > > Alas, they don't boot. :( > > > > Strange as I've had vanilla F-33 booting UEFI on a few devices via the > > images we got composing before GA and ImageFactory was about to boot > > them for the compose. > > Yeah, but that was likely newer kernel? 5.6 was GA for f32... > it's old at this point. :) > > Anyhow, I have reinstalled all the builders back to fedora 32 and > basically the config that they had before I upgraded them. :( > This should hopefully be stable for the mass rebuild. > > In the mean time I have filed: > https://bugzilla.redhat.com/show_bug.cgi?id=1920183 > on the OOM kojid killing. Hopefully we can figure it out and then I can > move them up after we do. Did those machines have zram, zswap, and/or normal swap enabled? (I looked at the dmesg attached in bugzilla, and it's only mentions zswap being loaded...) Zbyszek _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
