On Sun, Feb 3, 2019 at 6:19 AM Miroslav Suchý <msuchy@xxxxxxxxxx> wrote: > > Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a): > > - builds failing due to failure to download packages from official > > Fedora mirror dl.fedoraproject.org > > This is not first time I hear this. So I will open discussion to (again) alter builder's mock config to point to PHX > location, which is just rack away. > Wouldn't it make sense to maintain a local mirror of the distributions enabled in COPR and just have all the mock configs point to that? That eliminates all the network pressure and the issues we keep seeing there when trying to use it. > > - builds failing to import to COPR distgit (was not a problem before > > dist-git was introduced) > > This is actually new to me. Can you point me to some report? > I can't say I've experienced this issue. However, I do wish we could interact with COPR Dist-Git the same way we do with the official Fedora Dist-Git, with a Pagure instance and everything... > > - outages caused by read-only filesystem after reboot - copr is > > unusable until someone remounts it rw > > This was actually recently fixed. > > > - multi-day long outages caused by out of disk space > > - multi-hour or even day- long build queues > > That is because I actually care about the resources. It is very easy to submit a proposal and state that "it will run in > Fedora Cloud", but the cloud is not infinite. I can easily increase the soft limits in Cloud dashboard. But we are > either hitting hard limits (storage) or we would drastically limit other users using the same cloud (vCPU in case more > builders). > Copr is not designed for spikes, it is designed for the average traffic. Which is handling pretty well. You can check it > here: > https://copr.fedorainfracloud.org/status/stats/ > If you submit 300+ build today (like happened today) you will simply have to wait some time. At the same time, it should > not affect (too much) other users. This is IMO big difference from Koji which is IIRC strictly FIFO. > Would it be possible to extend COPR to support multiple locations for builders? For example, in addition to an OpenStack system, builders could be hosted on an oVirt system, or AWS, or GCP, and so on? That way it can support on-premise deployments and cloudy deployments too. Perhaps supporting even some limited form of auto-scaling for when it needs to "burst" to support more build traffic using clouds or hypervisors? For example, you could use AWS spot instances to do it on the cheap for the time a builder needs to run, and then have it go away afterward. I've done builds this way with buildbot and you can do thousands of builds for really cheap (on the order of hundreds of dollars each year). > > > - outdated SOP preventing non-COPR people like me from fixing COPR outages > > Feel free to raise any issue. Last update in this document was in October 2018. Checking it right now I see outdated > part about ppc builders, but otherwise it should be ok. > Thou, I will review that document after weekend. > I don't know anything about this... -- 真実はいつも一つ！/ Always, there's only one truth! _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
