On 2/4/19 4:03 PM, Neal Gompa wrote: > Aside from the times when it falls over for various reasons, I've had > entire days where I wait for a build to even start, because people who > use it for doing things like building KDE, chromium, or the Linux > kernel occupy literally all the available builder slots for a long > time. There aren't that many slots and it's easy to fill that up. > There's usually a large queue of packages to build, but not enough > builders to allow them to get done. Well, thats not supposed to be the case. Each user is given a limited amount of slots, so if say I dumped 100 kernels on it, it would only do a few of those at a time. So, sounds like a bug in allowing people more slots than they should have? > That indicates two things: > 1. The builders are weak and so builds take a long time (which means > slots are held up longer) Could be indeed. Could look at increasing the size... > 2. The demand and popularity of the service isn't being handled > appropriately (i.e. it should get more builders provisioned). Well, there's another issue here that seems to be a copr bug (see below) > I don't do things like build kernels often, but when I do, it usually > doesn't take all day. But stuff like Chromium is hard to build > locally, so I appreciate that we have somewhere to build and publish. > > But, as of right now, there are 16 tasks running, with 85 tasks > waiting for a builder. Yeah, but copr is allocated the following: 150 instances 200 vcps 488 GB ram So, either copr is loosing track of it's builders or openstack is doing something odd and not really providing that. I know openstack has accounting issues, but I do see about 65 or so builder instances, so I think this is on the copr side. So, fixing this would go a long way to helping out. Additionally, sometimes jobs finish, but copr shows them as still running. For example, the oldest job now: https://copr.fedorainfracloud.org/coprs/omos/kernel-testing/ which copr shows running for 11 hours, the logs show it's done (I don't know how long ago as there's no timestamps). > I wish we had visualizations like the ones OBS has[1][2], so that we > have an idea of how stuff is occupied and know at a glance that we're > over capacity. > > All I know right now is that it's easy to see that COPR gets into a > state where I just wind up waiting for builds to even start. > > [1]: https://build.opensuse.org/monitor > [2]: https://build.opensuse.org/monitor/old Well, we do have: https://copr.fedorainfracloud.org/status/stats/ but some of it doesn't match up with osbs, since copr builders are dynamic. kevin
Attachment:
signature.asc
Description: OpenPGP digital signature
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
