On Wed, Nov 13, 2019 at 1:24 PM Miro Hrončok <mhroncok@xxxxxxxxxx> wrote: > > On 13. 11. 19 18:31, Stephen Gallagher wrote: > > Sorry for the long overdue reply here. Answers to your questions are inline. > > Thank you. > > > On Wed, Oct 9, 2019 at 6:46 PM Miro Hrončok <mhroncok@xxxxxxxxxx> wrote: > > ... > >> What I miss in the description is: > >> > >> 1. How does this thing actually work? is there an additional repository composed > >> from the default streams available in Koji only? > > > > An additional repository is composed from the default streams > > (including the module metadata) and is merged with the standard > > buildroot using Koji's new "bare" mode. This buildroot will be > > available for non-Koji purposes (such as mock). > > Will this buildroot repository be mirrored or will it be more like the current > "local" repository defined in mock configs? > > For reference, this is currently in the rawhide mock config: > > [local] > name=local > baseurl=https://kojipkgs.fedoraproject.org/repos/rawhide/latest/$basearch/ > cost=2000 > enabled=0 > skip_if_unavailable=False > > My concern is that using the "local" repository for mock builds is quite slow > and IIRC puts Koji under unnecessary load. > The plan for the moment is to have it be the "local" repository. Given that it will at least start out fairly small (and not need deltarpms), we were going to see how well it held up. The longer-term plan will probably be to mirror it. > >> 2. How are conflicts between packages from the default streams and ursine > >> package be handled? > > > > As long as the module metadata is present, an artifact with a given > > name from an enabled or default module will effectively "hide" the > > non-modular version. > > I have a hard time to understand this. Could you please dive into the details? > > "As long"? What does that mean? When they are not present and when they are present? > Sorry, that was an English colloquialism. Basically I meant "unless we screw up and don't sync the metadata to the repository". So as long as a module stream providing an artifact with a given name is enabled or default, non-modular RPMs of the same name will be invisible to the solver. > "an enabled or default module"? Who enables a module in the buildroot? Is it > releng? The packager? Somebody else? Is it an explicit or implicit action? > "enabled" could happen as part of a mock config. That would be an explicit action on the part of the person creating the mock config. For the purposes of the discussion of Ursa Prime, you can ignore "enabled". I was trying to be thorough and ended up being confusing instead. > Suppose we put maven:3.5 into the buildroot (I know we don't do that, but that > was the idea origianlly). How do I as a a maintainer of an ursine maven-foo > package BuildRequire the ursine maven package? > You don't. Making a stream into the default stream is effectively making it equivalent to a non-modular package for most purposes. It already behaves this way for client installations; the content from a default stream is the only version DNF "sees" if it happens to override a non-modular package. On the client system, you'd need to `dnf module disable maven` to be able to pull down the non-modular package. Ursa Prime is basically a Change to have the Koji build environment behave the same way as the local environment would. This is why we've been discussing the need for the artifacts of default streams to behave *exactly* as if they were non-modular once delivered. Because they effectively are. They were built differently (though still in our infrastructure), but once built the requirement is that they must then behave like the non-modular distro would have. (Side-note: We retired the term "ursine" and replaced it with "non-modular" because it was confusing for non-native English speakers.) > > Unless you mean "conflicts" in some other sense, in which case I think > > RPM-level conflicts should handle it. > > You understood "conflicts" exactly as I've meant it here, sorry for not being > more explicit. > No problem, just wanted to make sure I was answering the correct question. > >> 3. What is the local experience if the packager is using mock. What if they are > >> using rpmbuild directly? > > > > For mock, if it's pointed at the Koji-hosted buildroot repository, it > > will work exactly as Koji does (with no local changes required). > > Makes sense, however, the information I miss is: Will it be pointed to the > Koji-hosted buildroot repository by default or not? My point was that this > should be part of the proposal before it was approved. > Yes, and you're right I will need to update the Change. > > If > > it's pointed at the public mirrors, it will need to have the Modular > > repos enabled and it will have whatever is in the standard buildroot > > plus default streams available there. > > Will it be pointed to the public mirrors with the Modular repos enabled? > It previously was like that for a while and it was reverted as a wrong behavior. > It wasn't reverted as "wrong behavior", it was reverted because libdnf at the time had a serious bug that mock was hitting. We never switched it back on, but it should work properly now. What we probably want to do is point it at the public mirrors, but have mirrormanager just point at Koji for now while we're proving it out. > > One purpose of this Change is to > > reconcile those two cases so they are the same. > > If the cases are the same, isn't it actually what was originally proposed as > Ursa Major? I am seriously confused here, because I understood this change as it > is explicitly doing a different thing, a nonmodular repository composed of the > modular content. So that our downstreams can consume it "traditionally" without > adapting their tooling. > What you're describing is Ursa Major. It was a plan to generate a non-modular repository with content from modules. FESCo rejected this outright because it required manual configuration to keep it working. We replaced it with Ursa Prime, which was built around the idea that we could track default streams and that would ensure things stayed in sync. In the repo created by Ursa Prime, it *does* carry modular metadata and relies on libdnf inside the mock environment to retrieve the correct packages. > What you are describing here sounds like we are doing Ursa Major, renamed to > Ursa Prime, because Ursa Prime was a no-go. Because I don't assume ill intent, I > am absolutely sure that there is something I'm not getting here, but it it very > hard to figure out what. > > Could you please, in great detail describe the differences between the two and > how this approved change addresses all the reasons that the previous attempt was > forbidden? > Well, the irony here is that throughout this conversation, we've basically settled on policy that would have made Ursa Major work just fine in Fedora... The big problem with Ursa Major is that we knew that it had a clear design flaw: it could not work in cases where two default streams depended on differing streams of another module. Ursa Major worked entirely by Koji tagging; it tagged certain builds into the buildroot directly. If you have a situation where two different module streams are both default and both carry a package with the same name, both would be tagged in and libdnf would end up picking whichever had the higher NVR (because it didn't have any of the module metadata to decide which one it actually needed). The secondary problem was that (partly because of the first problem), the tagging rules had to be maintained manually because we had no meaningful way to determine the presence of conflicts. We couldn't just script it to pull in all the defaults and call it a day, because of the known conflict problem. FESCo rejected Ursa Prime largely because infra asserted that maintaining that tagging list would be prohibitively difficult as Modularity became more ubiquitous. However, now that we've basically decided on "default streams may only depend on default streams", the original Ursa Major would have worked fine and been scriptable to just use the default streams. Ursa Prime is a completely different approach. Instead of trying to tag packages so that Koji's internal buildroot-generator includes their contents, we instead do a createrepo for the artifacts of the module default streams and then configure Koji to include that repository as an "external repo". This modular-buildroot external repo includes all of the modular metadata (unlike the internal Koji buildroot approach of Ursa Major) and thus mock can look up and install whatever packages are in the default streams or non-modular repository (so long as they aren't hidden by the default module version). That's what I meant above about how Ursa Prime makes it behave more like mock on the local system would work if you included the module repos and changed nothing else. They'll both use libdnf's logic to pull the appropriate packages. > > Right now, the Koji > > buildroot will be slightly different from the one mock would see > > because FESCo asked us to only allow two modules in the Koji buildroot > > for testing. > > Why would mock "see" different things based on restrictions FESCo asked to do? > My primary concern is that by default, mock sees what Koji sees. > > By this change and by what was approved by FESCo (to clarify: I voted against > this), we are adding new things to what Koji sees. So I am really not sure what > actually happens to what mock sees: > > 1. Do we add only the 2 FESCo approved modules to what mock sees? > 2. Do we add all default modular streams to what mock sees? > 3. Do we keep mock to only see ursine content? > > My opinion: > > 1. is ideal, but was not approved nor documented in the change proposal > 2. is not ideal and was not approved nor documented in the change proposal > 3. is not ideal, but is status quo, hence requires no approval/change > > > Prior to this Change, the Koji buildroot would have only > > the non-modular content. > > Correct. > > >> 4. How are we handling default streams with dependencies on non-default streams > >> of other modules? > > > > Throughout these discussions, it has been made clear that the best > > option for now will be to disallow default streams from depending on > > non-default streams in Fedora. This *can* (but may not) change in the > > future once we have worked out the explicit-vs-implicit enablement and > > upgrade questions. But for now, it seems reasonable to put that policy > > constraint on it. > > I agree. Let's make that happen **before** we actually start doing Ursa > Prime/Major/...? > > -- > Miro Hrončok > -- > Phone: +420777974800 > IRC: mhroncok > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
