On Wed, Jul 11, 2018 at 10:08 AM, Mikolaj Izdebski <mizdebsk@xxxxxxxxxx> wrote: > > On 07/11/2018 06:37 PM, Andrew Lutomirski wrote: > > From my perspective as an occasional Fedora packager, I'm regularly > > surprised by just how long it takes for Koji builders to install > > dependencies. I've never tried to dig in too far, but it looks like the > > builders download package metadata, download packages, and then install > > things. Surely this could be massively optimized by having the metadata > > pre-downloaded (at least when side tags aren't involved) and by having the > > packages already present over NFS or similar. > > Koji gets repodata and packages from HTTP servers, through caching > proxies located in the same datacenters as builders. Most often used > packages are cached in memory, so download speeds are not a problem. At > least for non-s390x builders. Accessing packages directly from NFS would > be slower. > I wonder if the time taken to decompress everything is relevant. Fedora currently uses xz, which isn't so fast. zchunk is zstd under the hood, which should be lot faster to decompress, especially on ARM builders. > > The slowest parts of setting up chroot is writing packages to disk, > synchronously. This part can be speeded up a lot by enabling nosync in > site-defaults.cfg mock config on Koji builders, setting cache=unsafe on > kvm buildvms, or both. These settings are safe because builders upload > all results to hubs upon task completion. With these settings chroot > setup can take about 30 seconds. I don't suppose this could get done? > > > Once this is optimized, another slow part is loading repodata into > memory - uncompressing it, parsing and creating internal libsolv data > structures. This could be speeded up by including solv/solvx files in > repodata, but I think that would require some code changes. Hmm. On my system, there are lots of .solv and .solvx files in /var/cache/dnf. I wonder if it would be straightforward to have a daily job that updates the builder filesystem by just having dnf refresh metadata and generate the .solv/.solvx files? There wouldn't be any dnf changes needed AFAICT -- just some management on the builder infrastructure. This would at least avoid a bunch of duplicate work on most builds. _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/message/JFLPBDPLIPH6WPH3LYUMV2BAB2O3C3ZJ/
