On Wed, 2020-04-08 at 15:19 -0700, Kevin Fenzi wrote: > On Wed, Apr 08, 2020 at 08:27:08PM +0000, Jeremy Cline wrote: > > Hi folks, > > > > The Fedora kernel is moving to maintaining the package in a source > > (sometimes people refer to it as an "exploded") tree. Basically > > just a > > fork of upstream. This makes a lot of packager tasks easier, but > > has > > introduced a minor issue with respect to the lookaside cache. > > > > Right now, it's configured to create a tarball from the git tree > > and > > upload it to the lookaside cache for each build. We build the > > rawhide > > kernel every weekday (give or take) and the xz compressed source > > tarball is ~110MB. This works out to about 28GB per year for > > Rawhide > > alone (if this is a drop in the bucket and no one cares please let > > me > > know and we'll just do this). The old approach uploaded a release > > tarball and then incremental tarballs on top of that. > > Well, the netapp will dedupe some of those blocks, but it would be > nice > to save some room I would think. > So the main concern with our current approach is it's wasting a bunch of space, but if you're not concerned I'm happy to not do any work and leave things as they are. > > If, however, Fedora allowed packagers to optionally generate > > tarballs > > from a git repository we could just push the linux git repository. > > The > > entire repository with history going back 15 years is under 4GB > > total, > > which is pretty good when compared to ~419GB which is the space > > required for the equivalent time using the lookaside cache. > > > > What would need to change: > > > > * Fedora offers a git repository to push source trees to. > > > > * A new file in the dist-git repository could be added if the > > packager > > wishes called "source-repos". In it, it contains a git url and > > commit > > identifier. For example, an entry might look like: > > " > > https://src.fedoraproject.org/sources/kernel.git v5.6" > > where v5.6 is a tag in the repository. We can restrict it so the > > git > > repository must be hosted by Fedora so we keep all the sources > > forever. > > > > * fedpkg and fedpkg-minimal would need to be updated to pull the > > source tree if the "source-repos" file is found and run > > "git archive". Fortunately this work is actually already done > > since > > Red Hat's version of fedpkg already supports this. > > > > I'm happy do to all the work for fedpkg/fedpkg-minimal to make this > > possible because the other option is to add a bunch of hacks to the > > kernel tooling to spit out a bunch of incremental tarballs to > > reduce > > what we have to upload. > > > > I assume this is something that will need to go through the > > packaging > > SIG, but from an infra side of things are there any > > thoughts/concerns? > > So, this was actually discussed already at some length: > > https://pagure.io/releng/issue/7498 > > Most of my concerns were answered, but then nim suggested this could > already be done with the macros we have today and no one said "no, > that > won't work for us" so I stopped doing anything until someone answered > and then it dropped off my radar. ;) > The macros would, I think, work for us, but we've actually got everything in a single repository and do a bunch of stuff (build valid configs for each arch, expand a templated specfile, etc) so we're just running git-archive locally rather than downloading it from the forge. So yeah, if no one's bothered about wasting a bunch of space with tarballs... - Jeremy _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
