On Wed, Nov 13, 2019 at 5:46 PM Miro Hrončok <mhroncok@xxxxxxxxxx> wrote: > > On 13. 11. 19 22:15, Stephen Gallagher wrote: ... > So the buildroot enabled modular repository contains only the 2 approved modules > including their modular metadata... > > >> "an enabled or default module"? Who enables a module in the buildroot? Is it > >> releng? The packager? Somebody else? Is it an explicit or implicit action? > >> > > > > "enabled" could happen as part of a mock config. That would be an > > explicit action on the part of the person creating the mock config. > > For the purposes of the discussion of Ursa Prime, you can ignore > > "enabled". I was trying to be thorough and ended up being confusing > > instead. > > ...and in Koji mock config, both are enabled and/or default? They will be marked as default in the modular-buildroot, yes. > > >> Suppose we put maven:3.5 into the buildroot (I know we don't do that, but that > >> was the idea origianlly). How do I as a a maintainer of an ursine maven-foo > >> package BuildRequire the ursine maven package? > >> > > > > You don't. > > > > Making a stream into the default stream is effectively making it > > equivalent to a non-modular package for most purposes. It already > > behaves this way for client installations; the content from a default > > stream is the only version DNF "sees" if it happens to override a > > non-modular package. On the client system, you'd need to `dnf module > > disable maven` to be able to pull down the non-modular package. > > I wonder what is the plan to handle this. > > I see only two proper ways out of here: > > 1) We only allow default modular streams if they only contain packages that are > retired from non-modular Fedora. Any request fro a default stream must be > coordinated with all the relevant package maintainers and those packages must be > retired from Fedora. > > 2) No default modular streams. (Yes, that again.) > > In my opinion, if we allow the non-modular packages to stay when there is a > default modular streams that "overrides" them, while effectively, our > maintainers cannot use the non-modular packages in the buildroot and our users > can only install the non-modular packages if they know how to disable a module, > it creates a very confusing situation. > You are correct. 1) is actually the policy[1] in Fedora today. There are a couple modules that are in violation of the policy, which is the cause of many of the issues we've been dealing with. The origin Specifically: "[I]f the module stream masks part of the Traditional RPM repos (e.g it replaces an existing RPM or it introduces a non-trivial set of conflicts) it may not be made a default stream without the express permission of FESCo. Unfortunately, the maven and ant module streams were made default before this policy (and the stronger one describing the api requirements) was in place. If this had been submitted to FESCo today, it would have been refused. [1] https://docs.fedoraproject.org/en-US/modularity/making-modules/managing-defaults/ > > Ursa Prime is basically a Change to have the Koji build environment > > behave the same way as the local environment would. > > > > This is why we've been discussing the need for the artifacts of > > default streams to behave *exactly* as if they were non-modular once > > delivered. Because they effectively are. They were built differently > > (though still in our infrastructure), but once built the requirement > > is that they must then behave like the non-modular distro would have. > > I hope we agree that they currently don't behave exactly like non-modular. > Correct, we've beaten that horse already and identified the key pieces that need to be fixed. > > (Side-note: We retired the term "ursine" and replaced it with > > "non-modular" because it was confusing for non-native English > > speakers.) > > Excellent choice. It was indeed rather confusing. Thank you. > > Side-side-note: In my opinion, "Ursa Major/Prime" is equally confusing. > You're not wrong. But we needed some way to refer to "The project for enabling the use of module artifacts in the non-modular buildroot" that didn't take fifteen seconds to say every time. These are essentially code-names, though. An end-user should never hear about them and once it is in place and actually works, packagers should not have to hear about it either. > >>>> 3. What is the local experience if the packager is using mock. What if they are > >>>> using rpmbuild directly? > >>> > >>> For mock, if it's pointed at the Koji-hosted buildroot repository, it > >>> will work exactly as Koji does (with no local changes required). > >> > >> Makes sense, however, the information I miss is: Will it be pointed to the > >> Koji-hosted buildroot repository by default or not? My point was that this > >> should be part of the proposal before it was approved. > >> > > > > Yes, and you're right I will need to update the Change. > > > >>> If > >>> it's pointed at the public mirrors, it will need to have the Modular > >>> repos enabled and it will have whatever is in the standard buildroot > >>> plus default streams available there. > >> > >> Will it be pointed to the public mirrors with the Modular repos enabled? > >> It previously was like that for a while and it was reverted as a wrong behavior. > >> > > > > It wasn't reverted as "wrong behavior", it was reverted because libdnf > > at the time had a serious bug that mock was hitting. We never switched > > it back on, but it should work properly now. > > I disagree. IIRC it was disabled because it didn't match the Koji behavior. > > https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/FNWYOOAUV73JUT3NKCZNW6TW66NGKJAK/ > > By "wrong behavior" I haven't meant "wrong forever", I meant "wrong at the moment". > > At this moment, *not having* the 2 approved modules in default mock config is > equally "wrong behavior". > OK, I misunderstood what you were saying. I think we're on the same page now. > > What we probably want to do is point it at the public mirrors, but > > have mirrormanager just point at Koji for now while we're proving it > > out. > > What I think we should do is to always try to match Koji as much as possible. > As long as Koji uses special repo, mock should use the same special repo or its > mirror. Not the entire modular repository. Not sure what repo exactly you meant > here by "public mirrors". > This would actually differ from how mock works with non-modular RPMs today. It points at the public mirrors of the Everything repo, not the Koji repo (unless you make manual edits to the mock config). This is why the original proposal was for Ursa Prime to just take whatever was in the defaults for the public Modular repo (plus the occasional override, when needed). So that it would be as close to "point mock at the public repos" as possible. I think that's the way we want this relationship to flow: that Koji's experience should match the public one. > >>> One purpose of this Change is to > >>> reconcile those two cases so they are the same. > > They were the same until this change. No modules in Koji, no modules in mock. > I mean that it was meant to ensure that if you added the Modular repos to mock, you would get the same *effective* experience as you would when you do a build in Koji. (As close to identical as you get today when pointed at the Everything repo rather than the Koji buildroot (aka "local") repo) > >> ... > > Well, the irony here is that throughout this conversation, we've > > basically settled on policy that would have made Ursa Major work just > > fine in Fedora... > > > > The big problem with Ursa Major is that we knew that it had a clear > > design flaw: it could not work in cases where two default streams > > depended on differing streams of another module. Ursa Major worked > > entirely by Koji tagging; it tagged certain builds into the buildroot > > directly. If you have a situation where two different module streams > > are both default and both carry a package with the same name, both > > would be tagged in and libdnf would end up picking whichever had the > > higher NVR (because it didn't have any of the module metadata to > > decide which one it actually needed). > > > > The secondary problem was that (partly because of the first problem), > > the tagging rules had to be maintained manually because we had no > > meaningful way to determine the presence of conflicts. We couldn't > > just script it to pull in all the defaults and call it a day, because > > of the known conflict problem. > > > > FESCo rejected Ursa Prime largely because infra asserted that > > maintaining that tagging list would be prohibitively difficult as > > Modularity became more ubiquitous. However, now that we've basically > > decided on "default streams may only depend on default streams", the > > original Ursa Major would have worked fine and been scriptable to just > > use the default streams. > > > > Ursa Prime is a completely different approach. Instead of trying to > > tag packages so that Koji's internal buildroot-generator includes > > their contents, we instead do a createrepo for the artifacts of the > > module default streams and then configure Koji to include that > > repository as an "external repo". This modular-buildroot external repo > > includes all of the modular metadata (unlike the internal Koji > > buildroot approach of Ursa Major) and thus mock can look up and > > install whatever packages are in the default streams or non-modular > > repository (so long as they aren't hidden by the default module > > version). That's what I meant above about how Ursa Prime makes it > > behave more like mock on the local system would work if you included > > the module repos and changed nothing else. They'll both use libdnf's > > logic to pull the appropriate packages. > > Thanks for explaining that difference. > Was it sufficiently clear? It's a complicated topic and I want to make sure I'm explaining it sufficiently. > >>> Right now, the Koji > >>> buildroot will be slightly different from the one mock would see > >>> because FESCo asked us to only allow two modules in the Koji buildroot > >>> for testing. > >> > >> Why would mock "see" different things based on restrictions FESCo asked to do? > >> My primary concern is that by default, mock sees what Koji sees. > >> Mock will only have access to either the koji modular-buildroot repo or the public Modular repo. In general, we probably want to prefer that people use the latter, but during the time period where we're only allowing a subset of the default streams into the buildroot, we have to point it at the koji modular-buildroot repo (or else it would be picking up all the default streams). > >> By this change and by what was approved by FESCo (to clarify: I voted against > >> this), we are adding new things to what Koji sees. So I am really not sure what > >> actually happens to what mock sees: > >> > >> 1. Do we add only the 2 FESCo approved modules to what mock sees? If we point mock at the koji modular-buildroot, then mock will only see what Koji sees. > >> 2. Do we add all default modular streams to what mock sees? If we point mock at the public, mirrored Modular repo, it will get all the default streams. > >> 3. Do we keep mock to only see ursine content? > >> > >> My opinion: > >> > >> 1. is ideal, but was not approved nor documented in the change proposal > >> 2. is not ideal and was not approved nor documented in the change proposal > >> 3. is not ideal, but is status quo, hence requires no approval/change > > Not sure if this was left intentionally out, or missed in my e-mail, so I'm > explicitly stating that this remained unanswered. > I missed this somehow. I'd say that we probably want to coordinate shipping a mock config update that points at the koji modular buildroot at the same time that we go live with Ursa Prime so that the Koji and mock experience remains the same. > >>>> 4. How are we handling default streams with dependencies on non-default streams > >>>> of other modules? > >>> > >>> Throughout these discussions, it has been made clear that the best > >>> option for now will be to disallow default streams from depending on > >>> non-default streams in Fedora. This *can* (but may not) change in the > >>> future once we have worked out the explicit-vs-implicit enablement and > >>> upgrade questions. But for now, it seems reasonable to put that policy > >>> constraint on it. > >> > >> I agree. Let's make that happen **before** we actually start doing Ursa > >> Prime/Major/...? > > Same here. I didn't interpret that as a question, since I'd pretty much agreed already. But sure. _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
