On Wed, 1 Jul 2020 at 08:09, Jan Kratochvil <jan.kratochvil@xxxxxxxxxx> wrote: > > On Wed, 01 Jul 2020 13:43:57 +0200, Stephen John Smoogen wrote: > > In the end we have limited resources and you are running into those > > limits. We can either have fewer builders with more disk space and a > > long queue of builds or we can have more builders with smaller disk > > space. > > Good to know disk space really is a limiting factor there (as it is in general > cheap compared to CPUs+RAM). > Disk space is only cheap if you don't care about the speed or the amount of usable disk space you have. [You can have a system with 200 TB of disk space from a bunch of 4TB sata disks but find that only 20 TB is speedy and the rest has a fall over in performance because none of the drives or controllers have enough cache] Every time we have tried to get larger disks, the amount of complaints for slow builders goes up exponentially. Trying for other disk layouts to increase IO ends up with other downtime complaints (oh look RAID-10 failed on 2 drives in the same pair.. there goes everything). Disk space which meets the IO needs of builders ends up being as expensive as CPU cores and RAM amounts. The faster the IO needs, the smaller the disk space per disk. You can fix this by having more disks, but the more disks per system, the higher the power and cooling costs per system. In the end, there is no such thing as a free lunch. > Sure a best step would be to have multiple (two) kinds of builders with > different disk sizes and blacklist/whitelist packages to them. I am aware that > idea is obvious, is there a Koji patch needed or even administration of two > kinds of builders is too much? > I believe we have done this in the past for the x86_64 arch. The s390, ppc64le, and aarch64 architectures have exceedingly limited resources to try this with. At the moment, we are still trying to bring up what we had in PHX2 to IAD2. At the moment, the sysadmin team for Fedora are running on fumes. We have worked several months of 10+ hour days+weekends to make the move a success and we will be working at least a month more to bring up the rest of the servers. At the moment we have about 10% of the hardware we shipped back in operation and there are at least 2 to 3 weeks to get the rest up. Getting up larger builders is going to have to wait. -- Stephen J Smoogen. _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
