On 13. 11. 19 22:15, Stephen Gallagher wrote:
On Wed, Nov 13, 2019 at 1:24 PM Miro Hrončok <mhroncok@xxxxxxxxxx> wrote:
On 13. 11. 19 18:31, Stephen Gallagher wrote:
Sorry for the long overdue reply here. Answers to your questions are inline.
Thank you.
On Wed, Oct 9, 2019 at 6:46 PM Miro Hrončok <mhroncok@xxxxxxxxxx> wrote:
...
What I miss in the description is:
1. How does this thing actually work? is there an additional repository composed
from the default streams available in Koji only?
An additional repository is composed from the default streams
(including the module metadata) and is merged with the standard
buildroot using Koji's new "bare" mode. This buildroot will be
available for non-Koji purposes (such as mock).
Will this buildroot repository be mirrored or will it be more like the current
"local" repository defined in mock configs?
My concern is that using the "local" repository for mock builds is quite slow
and IIRC puts Koji under unnecessary load.
The plan for the moment is to have it be the "local" repository. Given
that it will at least start out fairly small (and not need deltarpms),
we were going to see how well it held up. The longer-term plan will
probably be to mirror it.
OK. Since currently, nothing really depends on the modules added there, it makes
very little difference. I'm glad that there is a plan for the future.
2. How are conflicts between packages from the default streams and ursine
package be handled?
As long as the module metadata is present, an artifact with a given
name from an enabled or default module will effectively "hide" the
non-modular version.
I have a hard time to understand this. Could you please dive into the details?
"As long"? What does that mean? When they are not present and when they are present?
Sorry, that was an English colloquialism. Basically I meant "unless we
screw up and don't sync the metadata to the repository". So as long as
a module stream providing an artifact with a given name is enabled or
default, non-modular RPMs of the same name will be invisible to the
solver.
So the buildroot enabled modular repository contains only the 2 approved modules
including their modular metadata...
"an enabled or default module"? Who enables a module in the buildroot? Is it
releng? The packager? Somebody else? Is it an explicit or implicit action?
"enabled" could happen as part of a mock config. That would be an
explicit action on the part of the person creating the mock config.
For the purposes of the discussion of Ursa Prime, you can ignore
"enabled". I was trying to be thorough and ended up being confusing
instead.
...and in Koji mock config, both are enabled and/or default?
Suppose we put maven:3.5 into the buildroot (I know we don't do that, but that
was the idea origianlly). How do I as a a maintainer of an ursine maven-foo
package BuildRequire the ursine maven package?
You don't.
>
Making a stream into the default stream is effectively making it
equivalent to a non-modular package for most purposes. It already
behaves this way for client installations; the content from a default
stream is the only version DNF "sees" if it happens to override a
non-modular package. On the client system, you'd need to `dnf module
disable maven` to be able to pull down the non-modular package.
I wonder what is the plan to handle this.
I see only two proper ways out of here:
1) We only allow default modular streams if they only contain packages that are
retired from non-modular Fedora. Any request fro a default stream must be
coordinated with all the relevant package maintainers and those packages must be
retired from Fedora.
2) No default modular streams. (Yes, that again.)
In my opinion, if we allow the non-modular packages to stay when there is a
default modular streams that "overrides" them, while effectively, our
maintainers cannot use the non-modular packages in the buildroot and our users
can only install the non-modular packages if they know how to disable a module,
it creates a very confusing situation.
Ursa Prime is basically a Change to have the Koji build environment
behave the same way as the local environment would.
This is why we've been discussing the need for the artifacts of
default streams to behave *exactly* as if they were non-modular once
delivered. Because they effectively are. They were built differently
(though still in our infrastructure), but once built the requirement
is that they must then behave like the non-modular distro would have.
I hope we agree that they currently don't behave exactly like non-modular.
(Side-note: We retired the term "ursine" and replaced it with
"non-modular" because it was confusing for non-native English
speakers.)
Excellent choice. It was indeed rather confusing. Thank you.
Side-side-note: In my opinion, "Ursa Major/Prime" is equally confusing.
3. What is the local experience if the packager is using mock. What if they are
using rpmbuild directly?
For mock, if it's pointed at the Koji-hosted buildroot repository, it
will work exactly as Koji does (with no local changes required).
Makes sense, however, the information I miss is: Will it be pointed to the
Koji-hosted buildroot repository by default or not? My point was that this
should be part of the proposal before it was approved.
Yes, and you're right I will need to update the Change.
If
it's pointed at the public mirrors, it will need to have the Modular
repos enabled and it will have whatever is in the standard buildroot
plus default streams available there.
Will it be pointed to the public mirrors with the Modular repos enabled?
It previously was like that for a while and it was reverted as a wrong behavior.
It wasn't reverted as "wrong behavior", it was reverted because libdnf
at the time had a serious bug that mock was hitting. We never switched
it back on, but it should work properly now.
I disagree. IIRC it was disabled because it didn't match the Koji behavior.
https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx/thread/FNWYOOAUV73JUT3NKCZNW6TW66NGKJAK/
By "wrong behavior" I haven't meant "wrong forever", I meant "wrong at the moment".
At this moment, *not having* the 2 approved modules in default mock config is
equally "wrong behavior".
What we probably want to do is point it at the public mirrors, but
have mirrormanager just point at Koji for now while we're proving it
out.
What I think we should do is to always try to match Koji as much as possible.
As long as Koji uses special repo, mock should use the same special repo or its
mirror. Not the entire modular repository. Not sure what repo exactly you meant
here by "public mirrors".
One purpose of this Change is to
reconcile those two cases so they are the same.
They were the same until this change. No modules in Koji, no modules in mock.
...
Well, the irony here is that throughout this conversation, we've
basically settled on policy that would have made Ursa Major work just
fine in Fedora...
The big problem with Ursa Major is that we knew that it had a clear
design flaw: it could not work in cases where two default streams
depended on differing streams of another module. Ursa Major worked
entirely by Koji tagging; it tagged certain builds into the buildroot
directly. If you have a situation where two different module streams
are both default and both carry a package with the same name, both
would be tagged in and libdnf would end up picking whichever had the
higher NVR (because it didn't have any of the module metadata to
decide which one it actually needed).
The secondary problem was that (partly because of the first problem),
the tagging rules had to be maintained manually because we had no
meaningful way to determine the presence of conflicts. We couldn't
just script it to pull in all the defaults and call it a day, because
of the known conflict problem.
FESCo rejected Ursa Prime largely because infra asserted that
maintaining that tagging list would be prohibitively difficult as
Modularity became more ubiquitous. However, now that we've basically
decided on "default streams may only depend on default streams", the
original Ursa Major would have worked fine and been scriptable to just
use the default streams.
Ursa Prime is a completely different approach. Instead of trying to
tag packages so that Koji's internal buildroot-generator includes
their contents, we instead do a createrepo for the artifacts of the
module default streams and then configure Koji to include that
repository as an "external repo". This modular-buildroot external repo
includes all of the modular metadata (unlike the internal Koji
buildroot approach of Ursa Major) and thus mock can look up and
install whatever packages are in the default streams or non-modular
repository (so long as they aren't hidden by the default module
version). That's what I meant above about how Ursa Prime makes it
behave more like mock on the local system would work if you included
the module repos and changed nothing else. They'll both use libdnf's
logic to pull the appropriate packages.
Thanks for explaining that difference.
Right now, the Koji
buildroot will be slightly different from the one mock would see
because FESCo asked us to only allow two modules in the Koji buildroot
for testing.
Why would mock "see" different things based on restrictions FESCo asked to do?
My primary concern is that by default, mock sees what Koji sees.
By this change and by what was approved by FESCo (to clarify: I voted against
this), we are adding new things to what Koji sees. So I am really not sure what
actually happens to what mock sees:
1. Do we add only the 2 FESCo approved modules to what mock sees?
2. Do we add all default modular streams to what mock sees?
3. Do we keep mock to only see ursine content?
My opinion:
1. is ideal, but was not approved nor documented in the change proposal
2. is not ideal and was not approved nor documented in the change proposal
3. is not ideal, but is status quo, hence requires no approval/change
Not sure if this was left intentionally out, or missed in my e-mail, so I'm
explicitly stating that this remained unanswered.
4. How are we handling default streams with dependencies on non-default streams
of other modules?
Throughout these discussions, it has been made clear that the best
option for now will be to disallow default streams from depending on
non-default streams in Fedora. This *can* (but may not) change in the
future once we have worked out the explicit-vs-implicit enablement and
upgrade questions. But for now, it seems reasonable to put that policy
constraint on it.
I agree. Let's make that happen **before** we actually start doing Ursa
Prime/Major/...?
Same here.
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
