On Sat, 20 Nov 2021 at 15:05, Matthew Miller <mattdm@xxxxxxxxxxxxxxxxx> wrote: > > I mentioned on devel list that Sourcegraph is going to be indexing > source.fedoraproject.org. > > So I guess, first — heads-up! if you see their traffic, it's not malicious. > (I asked them to provide us with the user agent they use, and I'll pass that > on.) > > But then, I'd actually like to go a step further, and have them index _the > actual source for every build_. They're open to doing that, but what they > need is a git repo. > > So...... how hard / ridiculous / bad would it be to have a step in the build > process in koji which, between the %prep and %build phases, push the > unpacked-and-prepped source tree to Gitlab, under, say, > https://gitlab.com/fedora/exploded-build-sources/package-name/? > 0. When you mean %prep and %build you mean where koji builds a src.rpm and then the builders get it versus the prep/build stage of mock? 1. I don't think the builders don't have access to the internet to do that. Koji would have access but it is a limited resource. 2. I don't think you want any and every build to be broken because we could not get to gitlab for a push due to timeouts, problems with snips in the code etc. You would want to plumb this into the build process as a completely new tool. It would need a dedicated box which does this and it would need to be able to a) not stop builds and composes b) be able to queue these actions so a mass rebuild doesn't take weeks c) be able to resend/redo when we hit gitlab max actions per second/hour. [Even paid accounts have quotas to keep overall service working] d) deal with MBS issues. Otherwise you are going to run into various limitations that could break the koji+mbs+pdc+bodhi+composer+etc which are all built with certain tolerances and break when something slows things down. > I'm imaginging this would work something like this: > > 1. If remote repo doesn't exist, create it with the gitlab api > > 2. Do a shallow, no-checkout clone of that remote repo, using > --git-dir and --work-tree so that the .git directory isn't created inside > the working directory > > 3. copy the unpacked source tree from %prep into the work-tree dir > (are we building on btrfs? cp -l if not, to save io?) > > 4. git add --all > > 5. git commit -m "Build ID: ${buildID} https://koji.fedoraproject.org/koji/buildinfo?buildID=${buildID}"; > > 6. git push > > > With some details to be worked out. :) Like, repo tags as branches, maybe? > And, do this on all arches, or just one of them? Or maybe run up through > %prep as part of the src build rather than any of the binary builds? Make > the commit as the Fedora username of the person who did the build? > > But those kinds of implementation details aside -- does this seem like > something we might be able to do? > > Or, any ideas for an alternate approach? I mean, obviously, source-git would > be one such alternative, but getting that for _everything_ would require a > big rework of how everyone works. (I think we should get to that eventually, > but that's a long way off.) > > -- > Matthew Miller > <mattdm@xxxxxxxxxxxxxxxxx> > Fedora Project Leader > _______________________________________________ > infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure -- Stephen J Smoogen. Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren _______________________________________________ infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
