Re: Sourcegraph and exploded sources

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Sat, 20 Nov 2021 at 15:05, Matthew Miller <mattdm@xxxxxxxxxxxxxxxxx> wrote:
>
> I mentioned on devel list that Sourcegraph is going to be indexing
> source.fedoraproject.org.
>
> So I guess, first — heads-up! if you see their traffic, it's not malicious.
> (I asked them to provide us with the user agent they use, and I'll pass that
> on.)
>
> But then, I'd actually like to go a step further, and have them index _the
> actual source for every build_. They're open to doing that, but what they
> need is a git repo.
>
> So...... how hard / ridiculous / bad would it be to have a step in the build
> process in koji which, between the %prep and %build phases, push the
> unpacked-and-prepped source tree to Gitlab, under, say,
> https://gitlab.com/fedora/exploded-build-sources/package-name/?
>

0. When you mean %prep and %build you mean where koji builds a src.rpm
and then the builders get it versus the prep/build stage of mock?
1. I don't think the builders don't have access to the internet to do
that. Koji would have access but it is a limited resource.
2. I don't think you want any and every build to be broken because we
could not get to gitlab for a push due to timeouts, problems with
snips in the code etc.

You would want to plumb this into the build process as a completely
new tool. It would need a dedicated box which does this and it would
need to be able to
a) not stop builds and composes
b) be able to queue these actions so a mass rebuild doesn't take weeks
c) be able to resend/redo when we hit gitlab max actions per
second/hour. [Even paid accounts have quotas to keep overall service
working]
d) deal with MBS issues.

Otherwise you are going to run into various limitations that could
break the koji+mbs+pdc+bodhi+composer+etc which are all built with
certain tolerances and break when something slows things down.



> I'm imaginging this would work something like this:
>
> 1. If remote repo doesn't exist, create it with the gitlab api
>
> 2. Do a shallow, no-checkout clone of that remote repo, using
>    --git-dir and --work-tree so that the .git directory isn't created inside
>    the working directory
>
> 3. copy the unpacked source tree from %prep into the work-tree dir
>    (are we building on btrfs? cp -l if not, to save io?)
>
> 4. git add --all
>
> 5. git commit -m "Build ID: ${buildID} https://koji.fedoraproject.org/koji/buildinfo?buildID=${buildID}";
>
> 6. git push
>
>
> With some details to be worked out. :) Like, repo tags as branches, maybe?
> And, do this on all arches, or just one of them? Or maybe run up through
> %prep as part of the src build rather than any of the binary builds? Make
> the commit as the Fedora username of the person who did the build?
>
> But those kinds of implementation details aside -- does this seem like
> something we might be able to do?
>
> Or, any ideas for an alternate approach? I mean, obviously, source-git would
> be one such alternative, but getting that for _everything_ would require a
> big rework of how everyone works. (I think we should get to that eventually,
> but that's a long way off.)
>
> --
> Matthew Miller
> <mattdm@xxxxxxxxxxxxxxxxx>
> Fedora Project Leader
> _______________________________________________
> infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
> Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure



-- 
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard
battle. -- Ian MacClaren
_______________________________________________
infrastructure mailing list -- infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to infrastructure-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure




[Index of Archives]     [Fedora Development]     [Fedora Users]     [Fedora Desktop]     [Fedora SELinux]     [Yosemite News]     [KDE Users]

  Powered by Linux