FWIW, it looks like COPR's connectivity / download issues when installing the buildroot have gotten even worse over the past two days; The last 13 builds for my elementary-nightly repository **all** failed due to some download issue in root.log, and the situation was similar yesterday morning. At this point, COPR is becoming completely useless to catch upstream issues, since builds fail for COPR related issues 99% of the time, and only 1% are real issues. I usually try to re-submit builds until they either succeed or fail for real, but at this rate (and with the slow web interface), this isn't really feasible anymore. Fabio On Tue, Feb 5, 2019 at 6:07 AM Kevin Fenzi <kevin@xxxxxxxxx> wrote: > > On 2/4/19 4:03 PM, Neal Gompa wrote: > > > Aside from the times when it falls over for various reasons, I've had > > entire days where I wait for a build to even start, because people who > > use it for doing things like building KDE, chromium, or the Linux > > kernel occupy literally all the available builder slots for a long > > time. There aren't that many slots and it's easy to fill that up. > > There's usually a large queue of packages to build, but not enough > > builders to allow them to get done. > > Well, thats not supposed to be the case. Each user is given a limited > amount of slots, so if say I dumped 100 kernels on it, it would only do > a few of those at a time. So, sounds like a bug in allowing people more > slots than they should have? > > That indicates two things: > > 1. The builders are weak and so builds take a long time (which means > > slots are held up longer) > > Could be indeed. Could look at increasing the size... > > > 2. The demand and popularity of the service isn't being handled > > appropriately (i.e. it should get more builders provisioned). > > Well, there's another issue here that seems to be a copr bug (see below) > > > I don't do things like build kernels often, but when I do, it usually > > doesn't take all day. But stuff like Chromium is hard to build > > locally, so I appreciate that we have somewhere to build and publish. > > > > But, as of right now, there are 16 tasks running, with 85 tasks > > waiting for a builder. > > Yeah, but copr is allocated the following: > > 150 instances > 200 vcps > 488 GB ram > > So, either copr is loosing track of it's builders or openstack is doing > something odd and not really providing that. I know openstack has > accounting issues, but I do see about 65 or so builder instances, so I > think this is on the copr side. So, fixing this would go a long way to > helping out. > > Additionally, sometimes jobs finish, but copr shows them as still > running. For example, the oldest job now: > > https://copr.fedorainfracloud.org/coprs/omos/kernel-testing/ > > which copr shows running for 11 hours, the logs show it's done (I don't > know how long ago as there's no timestamps). > > > I wish we had visualizations like the ones OBS has[1][2], so that we > > have an idea of how stuff is occupied and know at a glance that we're > > over capacity. > > > > All I know right now is that it's easy to see that COPR gets into a > > state where I just wind up waiting for builds to even start. > > > > [1]: https://build.opensuse.org/monitor > > [2]: https://build.opensuse.org/monitor/old > > Well, we do have: > https://copr.fedorainfracloud.org/status/stats/ > but some of it doesn't match up with osbs, since copr builders are dynamic. > > kevin > > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
