On Sat, 29 Feb 2020 at 21:50, Nicolas Mailhot via devel
<devel@xxxxxxxxxxxxxxxxxxxxxxx> wrote:
>
> Le samedi 29 février 2020 à 20:30 +0100, clime a écrit :
> >
> > What about fedpkg local and fedpkg verrel then?
>
> Putting %{dynrel} reconciliation in the rpmbuild -bs stage using
> detached file state means that fedpkg local (or checking out git state
> and building in mock or copr or OBS or via plain rpmbuild -bs) will
> give you the same result as launching fedpkg build.
Well, I believe it doesn't. You run:
1) fedpkg local -> produces <somepkg>-1.0-1.fc32 (1 in release because
you haven't built that package before)
2) fedpkg build
3) vim <somepkg>.spec and do some change in %description
4) fedpkg commit -m "description improvement"
5) fedpkg local -> produces <somepkg>-1.0-1.fc32
6) fedpkg push -> error because build system pushed meanwhile
7) fedpkg pull --rebase (to quickly fix the error)
8) fedpkg build -> builds <somepkg>-1.0-2.fc32
To get the correct NVR for the package being built by `fedpkg local`
you need to wait for some external process to finish first so that you
can pull the new state. Only then you get correct value again.
> Which is exactly
> what you want to make QA and testing workflows work. Those don’t live
> exclusively in koji.
>
> And because only production builds get committed back the packager can
> change his mind and stage a few more changes before doing the
> production build, without polluting the changelog builds that were
> never pushed to users.
That's a good thing, definitely.
>
> For fedpkg verrel you'll probably want to output a last (saved in
> detached file) and next line (probably factorizable by externalizing
> the dynrel bump logic in a separate command). That’s more honest
> anyway, next may happen or not, when you launch fedpkg verrel, it’s
> mere potential (the next commit may bump version and invalidate your
> future build).
There is the same problem as above with `fedpkg local`, the command
won't be giving you correct values at all times.
>
> > As for changelog, generally with build system commiting back to
> > dist-git, there is a problem that I can be concurrently pushing
> > changes while the build system tries to push its changes and build
> > system is not human so it will not know how to resolve such
> > situation.
>
> I understand, but that’s the consequence of automating changelog. Right
> now the reconciliation is done by the human packager (you can get in a
> similar situation by working on a change at the time a mass rebuild
> runs).
It doesn't need to be the consequence of "automating changelog". It's
only a consequence in certain implementations. If the changelog is
derived purely from the git state, there aren't these problems
anymore. The mass rebuilds are easier to handle than your case because
with mass rebuilds, you always put the the new changelog record "on
top" of other changelogs records. I.e. when you cannot push from build
system (while peforming a mass rebuild), you can simply pull and try
to do the same thing as before. But in your proposal it's not that
easy (unless you do the locking or failing in that case as you
described below) because you would need modify/insert into "middle" of
the changelog at the right place.
>
> > Do you have a solution for situation when I launch a build and then I
> > do another change, commit, and push (the changes to changelog can be
> > quite arbitrary here)?
>
> You can set a lock at fedpkg build time, and forbid fedpkg commit in
> that branch till the lock is released (fedpkg build in the branch
> succeeded or not). The packager can still git commit things, as long as
> he does not touch the detached changelog file. Only fedpkg commit syncs
> git state with detached changelog state.
The fact that "git commit" is different from "fedpkg commit" is not
very convenient imho but could be probably get used to. But you
generally don't except commit actions to change the repository content
under your hands before actually committing. The locking mechanism you
describe can be easily work-arounded by cloning the package repo again
and doing your stuff. The only way to avoid it would be to implement
locking server side. Either way it's not the best user experience.
>
> An alternate simpler strategy would be to mark the fedpkg build as
> failure in koji if sync-back failed. That would work too. The build
> system need not be ultra smart, just robust WRT human behaviour.
I think not being able to work with dist-git while something is
building its content is a throughput limitation and doesn't bring the
best UX.
>
> (If the packager deliberately messes with the detached changelog while
> a production build for the same branch is ongoing he deserves a build
> failure – the changelog state is undefined till the build ends, so he’s
> doing changes on quicksand. If he tried that today he’d probably have
> to rewrite his changelog if the build failed)
>
> > I think, this is not a decision of rpm upstream but an infrastructure
> > thing or a mock thing.
>
> mock and rpm upstreams excel when they work together. My premise is
> that they are better qualified than us to do rpm/buildsys boundary fine
> tuning.
>
> Populating changelog from git and syncing back to fedora git are
> fedpkg/koji responsability, because Fedora git is Fedora specific
> infra. Handling release autobumping and build recording belongs to the
> lower layers, however. They’re the same with or without Fedora git,
> they belong to lower levels.
Yes, the updated %{dynrel} file and the updated changelog file can be
build products either of mock or rpmbuild.
>
> > I think your proposal wouldn't quite work for copr because it has no
> > access to those repositories (which especially for src.fp.o is good
> > in your proposal, otherwise copr would be modifying src.fp.o repos).
>
> copr does not need write access to Fedora git, its builds do not
> participate to the production build lineage as long as no one re-
> imports them in koji (which should be a deliberate fedpkg command, not
> something driven by copr itself).
>
> copr only needs access to its own private git to remember its own last
> successful build, if people want it to autobump from last successful
> state. (maybe they don’t). srpm import in copr will overwrite copr
> state with the state the new srpm contains, which is fine too.
>
> Putting state in detached srpm source files has the following super-
> duper property. You can import the srpm or scm checkout between
> systems, and they’ll just pick up from the point the previous system
> left things, without needing a full scm import.
That's nice if there is always just a single provider of the given
package in our ecosystem. In case of copr, you can fork a certain repo
and continue building there so you will be getting bumped numbers on
top of the work done in the original repo. But once the original repo
starts producing some builds as well you will be getting that package
in two versions in two different coprs and while the original repo
might have done less builds than the forked one since the fork
happened, it can contain the package with more features because the
forked repo was simply playing around.
I.e. as a user you might want to jump to the forked repo at some point
because there is some activity and you can update to the package from
that repo without any problems (because it builds upon the original
package) but once the original repo wakes up, you might want to jump
back and at that point, you need to downgrade. But this is a problem
that will be always there, just mentioning that situation like this
might happen. Not only NEVRAs play the role but their origin too.
I can see the advantage of your proposal that the auto-bumping
theoretically doesn't need git to work. But it still needs git because
you need push the updated %{dynstate} files somewhere. The only way to
avoid it would be to build a new srpm together with the rpm where the
srpm would be the same as the input srpm except for the updated
%{dynstate} files. So in case we wouldn't have src.fp.o and the
development would be fully decentralized, I could see your proposal to
be actually the only way to implement auto-bumping and it would be a
nice way. The thing is, the way we are developing today is different
from that and hence different solutions might possibly be more
suitable.
> So you do not need to
> deal with incompatible scms (or lack of scm).
Even if you create certain dedicated macros relying on git metadata
and put them into spec file, there should and will be a way to simply
export srpm from the repo which no longer has any git dependencies.
And then you can import that raw srpm anywhere else and continue
bumping manually or start using macros that work with the new SCM or
you can start using macros that don't rely on any SCM metadata but
rely on manually crafted external changelog (specified without the
headers but with new lines between separate changelog record groups),
mtime of spec file (you couldn't manually edit patches without
changing their names too in that case...but in the end, you could
count with mtimes of patch files too), and name-email file stored
alongside other files.
>
> > So if someone wanted to have a build counter in the release, yes,
> > this could be an implementation although it would be much easier to
> > simply catch the information from copr's database where this info is
> > stored
>
> It is simpler from the buildsys POW, but it ties state in a specific
> git and buildsystem. So it will break cross-buildsys workflows.
>
> Cross-buildsys workflows are critical for the project success because
> they enable sharing work with other distros, and allow packagers to
> make the best of a palette of build systems (each with its own
> constrains and limitiations). Fun fact: I noticed today than one of my
> old Fedora packages was rebuilt by others for AIX. This kind of cross-
> pollination is one of the strengths of the rpm ecosystem. Don’t break
> it by making out packages depend on Fedora git or buildsys state.
Well, only automatic versioning would be dependent on Fedora git or
buildsys state and as I tried to say earlier (downstream) versioning
is not universal across the whole rpm universe. But even versioning
wouldn't be actually dependent on anything because you always have a
way to "export" things as they currently are and import them elsewhere
(true, in case the release would be derived from some value stored in
copr's db, you would need to manually set the latest dynamic release
copr has produced into the spec file you are exporting or manually
create the %{dynrel} file before importing it somewhere else).
>
> > in fact there would never be a changelog file at
> > all because nobody uses fedpkg commit in git repos that are not from
> > src.fp.o so this dynamic changelog thing wouldn't be usable for copr
> > at all.
>
> When using a buildsys that didn’t automate changelog filling, you can
> just write in the detached changelog file directly. The build part
> won’t care how you filled this file with changes, it just needs it to
> record a build point.
The point is that COPR could and potentially would like to implement
the changelog filling but even if it does implement it, there is no
way to push the resulting changelog back to the source repo, it could
only push it back into its "private" repo but then user would need to
manually take that file and put it into his/her source repo manually
which is very inconvenient and it's not an automatic changelog
generation one would wish for. So basically you are saying, such
external repository can have automated changelog but it would need to
be generated by some custom tooling and this changelog file would need
to be complete with all the timestamps, emails, and verrels in
headers. So In other words, for copr+external repo we would have
something else than for koji+fedora dist-git.
>
> scm to changelog plumbing logic is going to change over time (for
> example, if we drop pagure for gitlab, or git for the next best thing).
> It is dangerous to set it in stone.
It is, however, good to have it implemented only on one place in a way
that it is easily reusable.
---
It's great that somebody can come up with such a different approach
than what was originally suggested.
Best regards and thanks!
clime
>
> dynrel autobumping and recording builds in the changelog is more of an
> rpm format change. It can (and should) be done in an independant way.
>
> Regards,
>
> --
> Nicolas Mailhot
> _______________________________________________
> devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
