On Sun, 3 Feb 2019 at 16:43, Neal Gompa <ngompa13@xxxxxxxxx> wrote: > > On Sun, Feb 3, 2019 at 6:19 AM Miroslav Suchý <msuchy@xxxxxxxxxx> wrote: > > > > Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a): > > > - builds failing due to failure to download packages from official > > > Fedora mirror dl.fedoraproject.org > > > > This is not first time I hear this. So I will open discussion to (again) alter builder's mock config to point to PHX > > location, which is just rack away. > > So problem 1 is that dl.fedoraproject.org is 1 rack away. There is some sort of problem with the setup of the dl servers which is no longer using memory efficiently. I think it needs a local varnish or similar caching server but i haven't had time to analyze. > > Wouldn't it make sense to maintain a local mirror of the distributions > enabled in COPR and just have all the mock configs point to that? That > eliminates all the network pressure and the issues we keep seeing > there when trying to use it. > COPR doesn't have the disk space to mirror all that at speed. [Yes you can cheaply stick them on slow large drives but trying to run the several dozen builders would be worse than going across the rack to the other network server. So you then need a lot of slow large drives and then you are no longer cheap and it seems like you have a lot of empty disk space that isn't being used so you might as well go with smaller expensive SAS/SSD. ] > > Would it be possible to extend COPR to support multiple locations for > builders? For example, in addition to an OpenStack system, builders > could be hosted on an oVirt system, or AWS, or GCP, and so on? That > way it can support on-premise deployments and cloudy deployments too. > I expect it is possible but it is going to come down to a credential/funding problem. Going out to the cloud like GCP/AWS requires funding and setup to make sure that the systems aren't compromised. [We have had several cloud offers from smaller vendors but they then wanted our root password and other credentials.. just to remind you that this isn't your hardware and they can look in it any time you want.] There would also be the speed issue of getting the data from the on-premise to the cloud. That takes time and sometimes seems to take longer than waiting in queue locally. Then there is the mirroring part of the data. [That seems to be slower than molasses at times on the order of a day. I expect it is partially that we are doing something naive and would be better dealt with elsewhere.] > Perhaps supporting even some limited form of auto-scaling for when it > needs to "burst" to support more build traffic using clouds or > hypervisors? For example, you could use AWS spot instances to do it on > the cheap for the time a builder needs to run, and then have it go > away afterward. I've done builds this way with buildbot and you can do > thousands of builds for really cheap (on the order of hundreds of > dollars each year). > That would be useful. -- Stephen J Smoogen. _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
