Nice, I have been trying to fight through the 'git context already missing' with pure lua rpm macros,
and so far was hitting walls left and right :-)
Will look at https://pagure.io/rpkg-util, might have more questions :-)
Adam
On Tue, Feb 25, 2020 at 12:20 AM clime <clime@xxxxxxxxxxxxxxxxx> wrote:
Hello!
On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon <pingou@xxxxxxxxxxxx> wrote:
>
> Good Morning Everyone,
>
> This topic has already been discussed a few times over the past month, but Adam
> Saleh, Nils Philippsen and myself have had the opportunity to invest some time
> on it with the hope of making the packager's life simpler as well as making it
> easier to build automation around our package maintenance.
>
> We have investigated a few ideas, the full list with their pros and cons can be
> found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev
> If you have any questions about some of these, please ask them, we'll be happy
> to answer them and potentially complete this document.
>
>
> For both the release and the changelog fields we believe using RPM macros would
> satisfy the requirements we have for opting-in/out:
> - You can easily opt-in by using the macros
> - You can easily opt-out by removing the macros from your spec file
>
>
> For the changelog, we believe we have a potential good solution:
> - The changelog will be automatically generated using an external `changelog`
> file as well as the commit history
> - When you opt-in, you will simply move the existing changelog to an external
> file `changelog`, which is placed in the dist-git repo and add the
> appropriate macro.
> - Upon building, the macro will generate the changelog using all the commits
> of the repo up to the last commit touching the `changelog` file. Of all
> these commits it will only consider the commits following these rules:
> - There are generally two approaches regarding what to include by default:
> 1. commits touching only the `sources`, `.gitignore`, `dead.package`
> files, `tests` folder will be ignored by default, i.e. a blacklist
> 2. only commits touching the spec file or patches will be included by
> default, i.e. a whitelist
> ==> Which approach do you think is better/will work in most cases?
> - empty commits (not touching any files) will be included on the assumption
> that this is their purpose
> - commits containing "magic keyword" (#changelog_exclude,
> #changelog_include?) will be ignored or included as the case may be
> - Finally the content of the changelog file specified will be appended to the
> changelog generated from the git history
>
>
> If you wanted to edit the changelog, you would:
> - Generate the changelog locally via a command like:
> `fedpkg generate-changelog > changelog`
> - Edit `changelog` as desired
> - Commit and push the changes
>
> Since the macro will only consider the commits up to the first commit touching
> `changelog` (in other words, the macro will only consider the commits after this
> one) and include `changelog` file as is, your edits will appear in the RPM
> changelog as you want.
>
> One thing to note is that rebuilds from the same commit will have the same
> %changelog, they will not get a new entry. If you want a new changelog entry,
> you have to create a new commit with the desired changelog entry as commit
> message (the commit itself can be empty).
>
>
>
> However, for the release field, we are struggling a little bit more, two options
> are more appealing to us:
>
> A) The release field is automatically generated using two elements:
> - the number of commits at this version
> - the number of builds at this commit
> - the dist-tag being added after them
> The release field would thus look like:
> ``<number of commit at version>.<number of build at commit>%{dist}``
>
> So we could have: (A, B, C and D being different commits)
> A -- version: 0.9 -- release: ?
> |
> B -- version: 1.0 -- release: 1.0
> |
> C -- version: 1.0 -- release: 2.0
> |
> D -- version: 1.1 -- release: 1.0
>
> A rebuild of the commit D, would lead to a release 1.1 (assuming the first one
> succeeded)/
>
> Pros:
> - Easy to replicate locally
> - Every changelog entry can be mapped to a `version-release` (No changes to the
> packaging guidelines)
> - Allows triggering two builds from the same commit without any manual change
> (simplifies mass-rebuilds)
> - Easy to link a certain build with a specific commit
> - Easy to “guess” the next release value before triggering the build
>
> Cons:
> - Old packages which are no longer receiving upstream releases may need
> special care (for example if they are at the release -48 but only had 45
> commits leading to this)
> - Relies on information from the build system for the build number (but can
> be closely simulated locally since the <n_build> info is only used for
> rebuilds)
> - Upgrade path may be problematic if Fn is upgraded to version X with one
> commit while Fn-1 is upgrade to version X with 2 commits (or more)
>
>
> B) The release field is automatically generated taking existing builds in all
> current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for
> builds of the same epoch:version, find a new release that (if at all possible)
> ensures upgradability from older Fedora versions to newer ones, adhering to the
> structure of a release tag documented in the Versioning Guidelines[1]. Going
> from the (RPM-wise) "latest build" that the new one should surpass, this can
> mean bumping in the front (`pkgrel`) or the back (`minorbump`).
>
> This allows packages from "very stable" upstreams who haven't released in years
> to still benefit from automatically generated releases.
>
> The following examples would use a macro for the release field as outlined in a
> separate document[2].
>
> Example #1 ("normal" release progression):
> Rawhide has: 2.0-1.fc32
> F31 has: 1.0-1.fc31
> F30 has: 1.0-1.fc30
>
> Next release in F31: 1.0-2.fc32
>
>
> Example #2 ("hotfix" in an older release, selected by an alternative macro (or
> option) in the spec file):
> Rawhide has: 2.0-1.fc32
> F31 has: 1.0-1.fc31
> F30 has: 1.0-1.fc30
>
> Next release in F30: 1.0-1.fc30.1
>
> Example #3 (pre-release, selected by an alternative macro (or option) in the
> spec file):
> Rawhide has: 2.0-1.fc32
> F31 has: 1.0-1.fc31
> F30 has: 1.0-1.fc30
>
> Next release in Rawhide: 3.0-0.1.20200224git1234abcd
>
>
> Pros:
> - Allows triggering two builds from the same commit without manual
> intervention
> - Emulates what many maintainers do manually today for most use cases
> - Can cater for pre-releases (e.g.: by offering different macros or macro
> options for the different use cases)
>
> Cons:
> - Needs the build system for information about builds in this and other Fedora
> versions
> - Not easy to reproduce locally because we don’t have machine-consumable
> knowledge about other builds in e.g. the dist-git repo
> - Does not allow to generate changelog entries with `version-release`
> information for all commits (and this will require a change in our packaging
> guidelines)
>
>
> So these are the results of our current investigations, we are very much eager
> to get your feedback on them and even more eager if you have ideas on how to
> approach/solve some of the challenges mentioned here.
>
>
> Looking forward for the discussion,
>
> Pierre
> For Adam, Nils and myself
What is the point of including number of builds into release? I think
the Miro's approach solves it.
Or is there any other problem except soname bumps?
Ad. document - annotated git tags:
(-) Editing the changelog would mean allowing to remove the git tags,
thus leading to potential issue in build reproducibility
That doesn't need to be the case. In rpkg-util, this was resolved by
introducing arguments since_tag and until_tag
for git_changelog macro
(https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros).
That's
how it can be prevented for some annotated tag to contribute to changelog.
Example:
{{{ git_changelog since_tag=name-1.3-1 }}}
* Mon Feb 24 2020 clime <clime@xxxxxxxxxxxxxxxxx> 1.2-1
- manual changelog entry that is used instead of a tag annotation
{{{ git_changelog until_tag=name-1.1-1 }}}
Removing already pushed git tags is probably not a good idea anyway:
https://git-scm.com/docs/git-tag#_on_re_tagging
Ad. the following approach for calculating release:
- Compute the release field from the number of commits since the last
version change: <version>-<commits_number>%{dist}
I think it is a good idea. In rpkg-util, it is done similarly, except
the release part has more subparts than just
two (including %{dist}).
The full description is here:
https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262
but the main difference is that it counts commits from the latest
annotated tag which contains (in its name)
the current version and the current release number which are extracted
from the spec file when
creating the tag unless they are specified manually on command line.
Commit count is only appended
to it if we build from commit which is on top of some annotated tag
(i.e. it is itself untagged).
Going by just a textual version change in a spec file might be slightly tricky.
You would need to go through all the past commits that touched that spec file,
keep checking these out and look if the version is changed when compared to the
one parsed from the head commit. Possible though.
To go back to your original post:
> For both the release and the changelog fields we believe using RPM macros would
> satisfy the requirements we have for opting-in/out:
By using such RPM macros, you would lose ability to rebuild srpms
because it will
be only possible to build them when the git context is present but not when they
become a standalone thing. I.e. things like this will not work:
https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mockbuild/backend.py#L270
That's why I think that such macros should be of a different kind:
macros that are computed
once when srpm is created and result of which is put _verbatim_ into
the spec file so that
there is no (re)computation later when srpm is being built and when
the git context is already
missing.
This approach is taken in the rpkg-util project
(https://pagure.io/rpkg-util). I really
recommend looking at it as I spent more than a year solving this
particular problem with
changelog and release (but actually not only that problem) and I also
optimized the macros there
as much as I possibly could.
If you want to play around with it, you can download the latest
version from here:
https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/
and try it on this:
https://pagure.io/hello_rpkg
clime
>
>
> [1]: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/
> [2]: https://hackmd.io/kuXOPe74RfepuztBz7pBsg
> _______________________________________________
> devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
> To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
_______________________________________________
devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
_______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
