So, I thought I would provide an update here. The mainframe admins discovered a large part of our woes: Our lpars were on "slow" 7200 rpm disk. They migrated us to "fast" local disk a few weeks ago, and indeed things are MUCH better from what I can see. The problem is not 100% solved, as I see we still have a lot of 'stolen' cpu cycles (ie, there are higher priority lpars running and they get cpus and we have to wait longer for them). This issue will hopefully be solved late this year/early next when more cpu resources are available. In the mean time, I think things should be stable now. With the i/o issues fixed we shouldn't see those download errors or failed unpacking rpms or the like. If you all see any new errors, please do file an infrastructure ticket. thanks! kevin -- On Fri, Aug 28, 2020 at 10:48:08AM -0700, kevin wrote: > Greetings. > > As many of you know, the s390x builders have been very slow or failing > builds with intermittent i/o issues for a while now. > > I've done what I can to mitigate this on the builder level, but the > problem is at a deeper level. > > I've been asked to try and collect issues that package maintainers are > hitting on these builders and provide them to mainframe admins so they > can understand the impact on us and prioritize more resources or other > corrective measures. > > To that end, if you are a maintainer and: > > * A build on s390x being slow affects you (needed for another build, > important bug fix, etc) in a serious way > > or > > * a build on a s390x builder fails in some odd way that is NOT related > to your package (unable to download src.rpm, checksum mismatches, etc). > > I'd love for you to note: > > * the link to the failed/slow task > * The time (UTC preferred) > * which exact builder it was > * what the issue was > * how it impacted your fedora work > > into: https://pagure.io/fedora-infrastructure/issue/9232 > > Please don't post to the list, mail me privately, etc. > Just add the info to the above issue. > > Thanks for your help > > kevin > _______________________________________________ > devel-announce mailing list -- devel-announce@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-announce-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel-announce@xxxxxxxxxxxxxxxxxxxxxxx > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
Attachment:
signature.asc
Description: PGP signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
