Hello Adam! On Tue, 25 Feb 2020 at 08:58, Adam Saleh <asaleh@xxxxxxxxxx> wrote: > > Nice, I have been trying to fight through the 'git context already missing' with pure lua rpm macros, > and so far was hitting walls left and right :-) > > Will look at https://pagure.io/rpkg-util, might have more questions :-) You have probably already noticed...the docs at https://docs.pagure.org/rpkg-util/index.html are for rpkg-util version that is currently in Fedora but it contains some nice introduction, nevertheless. The (newer) git version from the pagure repository has some features (e.g. git_release macro), which are not yet documented there. man page of the git version ($ man rpkg) is up to date, however. Thank you for looking at it. clime > > Adam > > On Tue, Feb 25, 2020 at 12:20 AM clime <clime@xxxxxxxxxxxxxxxxx> wrote: >> >> Hello! >> >> On Mon, 24 Feb 2020 at 17:50, Pierre-Yves Chibon <pingou@xxxxxxxxxxxx> wrote: >> > >> > Good Morning Everyone, >> > >> > This topic has already been discussed a few times over the past month, but Adam >> > Saleh, Nils Philippsen and myself have had the opportunity to invest some time >> > on it with the hope of making the packager's life simpler as well as making it >> > easier to build automation around our package maintenance. >> > >> > We have investigated a few ideas, the full list with their pros and cons can be >> > found in this document: https://hackmd.io/8pSwJidhTYel9euQqMEKpw?viev >> > If you have any questions about some of these, please ask them, we'll be happy >> > to answer them and potentially complete this document. >> > >> > >> > For both the release and the changelog fields we believe using RPM macros would >> > satisfy the requirements we have for opting-in/out: >> > - You can easily opt-in by using the macros >> > - You can easily opt-out by removing the macros from your spec file >> > >> > >> > For the changelog, we believe we have a potential good solution: >> > - The changelog will be automatically generated using an external `changelog` >> > file as well as the commit history >> > - When you opt-in, you will simply move the existing changelog to an external >> > file `changelog`, which is placed in the dist-git repo and add the >> > appropriate macro. >> > - Upon building, the macro will generate the changelog using all the commits >> > of the repo up to the last commit touching the `changelog` file. Of all >> > these commits it will only consider the commits following these rules: >> > - There are generally two approaches regarding what to include by default: >> > 1. commits touching only the `sources`, `.gitignore`, `dead.package` >> > files, `tests` folder will be ignored by default, i.e. a blacklist >> > 2. only commits touching the spec file or patches will be included by >> > default, i.e. a whitelist >> > ==> Which approach do you think is better/will work in most cases? >> > - empty commits (not touching any files) will be included on the assumption >> > that this is their purpose >> > - commits containing "magic keyword" (#changelog_exclude, >> > #changelog_include?) will be ignored or included as the case may be >> > - Finally the content of the changelog file specified will be appended to the >> > changelog generated from the git history >> > >> > >> > If you wanted to edit the changelog, you would: >> > - Generate the changelog locally via a command like: >> > `fedpkg generate-changelog > changelog` >> > - Edit `changelog` as desired >> > - Commit and push the changes >> > >> > Since the macro will only consider the commits up to the first commit touching >> > `changelog` (in other words, the macro will only consider the commits after this >> > one) and include `changelog` file as is, your edits will appear in the RPM >> > changelog as you want. >> > >> > One thing to note is that rebuilds from the same commit will have the same >> > %changelog, they will not get a new entry. If you want a new changelog entry, >> > you have to create a new commit with the desired changelog entry as commit >> > message (the commit itself can be empty). >> > >> > >> > >> > However, for the release field, we are struggling a little bit more, two options >> > are more appealing to us: >> > >> > A) The release field is automatically generated using two elements: >> > - the number of commits at this version >> > - the number of builds at this commit >> > - the dist-tag being added after them >> > The release field would thus look like: >> > ``<number of commit at version>.<number of build at commit>%{dist}`` >> > >> > So we could have: (A, B, C and D being different commits) >> > A -- version: 0.9 -- release: ? >> > | >> > B -- version: 1.0 -- release: 1.0 >> > | >> > C -- version: 1.0 -- release: 2.0 >> > | >> > D -- version: 1.1 -- release: 1.0 >> > >> > A rebuild of the commit D, would lead to a release 1.1 (assuming the first one >> > succeeded)/ >> > >> > Pros: >> > - Easy to replicate locally >> > - Every changelog entry can be mapped to a `version-release` (No changes to the >> > packaging guidelines) >> > - Allows triggering two builds from the same commit without any manual change >> > (simplifies mass-rebuilds) >> > - Easy to link a certain build with a specific commit >> > - Easy to “guess” the next release value before triggering the build >> > >> > Cons: >> > - Old packages which are no longer receiving upstream releases may need >> > special care (for example if they are at the release -48 but only had 45 >> > commits leading to this) >> > - Relies on information from the build system for the build number (but can >> > be closely simulated locally since the <n_build> info is only used for >> > rebuilds) >> > - Upgrade path may be problematic if Fn is upgraded to version X with one >> > commit while Fn-1 is upgrade to version X with 2 commits (or more) >> > >> > >> > B) The release field is automatically generated taking existing builds in all >> > current Fedora releases (ie: rawhide, Fn, Fn-1...) into account. This means for >> > builds of the same epoch:version, find a new release that (if at all possible) >> > ensures upgradability from older Fedora versions to newer ones, adhering to the >> > structure of a release tag documented in the Versioning Guidelines[1]. Going >> > from the (RPM-wise) "latest build" that the new one should surpass, this can >> > mean bumping in the front (`pkgrel`) or the back (`minorbump`). >> > >> > This allows packages from "very stable" upstreams who haven't released in years >> > to still benefit from automatically generated releases. >> > >> > The following examples would use a macro for the release field as outlined in a >> > separate document[2]. >> > >> > Example #1 ("normal" release progression): >> > Rawhide has: 2.0-1.fc32 >> > F31 has: 1.0-1.fc31 >> > F30 has: 1.0-1.fc30 >> > >> > Next release in F31: 1.0-2.fc32 >> > >> > >> > Example #2 ("hotfix" in an older release, selected by an alternative macro (or >> > option) in the spec file): >> > Rawhide has: 2.0-1.fc32 >> > F31 has: 1.0-1.fc31 >> > F30 has: 1.0-1.fc30 >> > >> > Next release in F30: 1.0-1.fc30.1 >> > >> > Example #3 (pre-release, selected by an alternative macro (or option) in the >> > spec file): >> > Rawhide has: 2.0-1.fc32 >> > F31 has: 1.0-1.fc31 >> > F30 has: 1.0-1.fc30 >> > >> > Next release in Rawhide: 3.0-0.1.20200224git1234abcd >> > >> > >> > Pros: >> > - Allows triggering two builds from the same commit without manual >> > intervention >> > - Emulates what many maintainers do manually today for most use cases >> > - Can cater for pre-releases (e.g.: by offering different macros or macro >> > options for the different use cases) >> > >> > Cons: >> > - Needs the build system for information about builds in this and other Fedora >> > versions >> > - Not easy to reproduce locally because we don’t have machine-consumable >> > knowledge about other builds in e.g. the dist-git repo >> > - Does not allow to generate changelog entries with `version-release` >> > information for all commits (and this will require a change in our packaging >> > guidelines) >> > >> > >> > So these are the results of our current investigations, we are very much eager >> > to get your feedback on them and even more eager if you have ideas on how to >> > approach/solve some of the challenges mentioned here. >> > >> > >> > Looking forward for the discussion, >> > >> > Pierre >> > For Adam, Nils and myself >> >> What is the point of including number of builds into release? I think >> the Miro's approach solves it. >> Or is there any other problem except soname bumps? >> >> Ad. document - annotated git tags: >> (-) Editing the changelog would mean allowing to remove the git tags, >> thus leading to potential issue in build reproducibility >> >> That doesn't need to be the case. In rpkg-util, this was resolved by >> introducing arguments since_tag and until_tag >> for git_changelog macro >> (https://docs.pagure.org/rpkg-util/macro_reference.html#git-macros). >> That's >> how it can be prevented for some annotated tag to contribute to changelog. >> >> Example: >> >> {{{ git_changelog since_tag=name-1.3-1 }}} >> >> * Mon Feb 24 2020 clime <clime@xxxxxxxxxxxxxxxxx> 1.2-1 >> - manual changelog entry that is used instead of a tag annotation >> >> {{{ git_changelog until_tag=name-1.1-1 }}} >> >> Removing already pushed git tags is probably not a good idea anyway: >> https://git-scm.com/docs/git-tag#_on_re_tagging >> >> Ad. the following approach for calculating release: >> - Compute the release field from the number of commits since the last >> version change: <version>-<commits_number>%{dist} >> >> I think it is a good idea. In rpkg-util, it is done similarly, except >> the release part has more subparts than just >> two (including %{dist}). >> >> The full description is here: >> https://pagure.io/rpkg-util/blob/master/f/man/rpkg_man_page.py#_262 >> but the main difference is that it counts commits from the latest >> annotated tag which contains (in its name) >> the current version and the current release number which are extracted >> from the spec file when >> creating the tag unless they are specified manually on command line. >> Commit count is only appended >> to it if we build from commit which is on top of some annotated tag >> (i.e. it is itself untagged). >> >> Going by just a textual version change in a spec file might be slightly tricky. >> You would need to go through all the past commits that touched that spec file, >> keep checking these out and look if the version is changed when compared to the >> one parsed from the head commit. Possible though. >> >> To go back to your original post: >> > For both the release and the changelog fields we believe using RPM macros would >> > satisfy the requirements we have for opting-in/out: >> >> By using such RPM macros, you would lose ability to rebuild srpms >> because it will >> be only possible to build them when the git context is present but not when they >> become a standalone thing. I.e. things like this will not work: >> >> https://github.com/rpm-software-management/mock/blob/cfe34c8a57/mock/py/mockbuild/backend.py#L270 >> >> That's why I think that such macros should be of a different kind: >> macros that are computed >> once when srpm is created and result of which is put _verbatim_ into >> the spec file so that >> there is no (re)computation later when srpm is being built and when >> the git context is already >> missing. >> >> This approach is taken in the rpkg-util project >> (https://pagure.io/rpkg-util). I really >> recommend looking at it as I spent more than a year solving this >> particular problem with >> changelog and release (but actually not only that problem) and I also >> optimized the macros there >> as much as I possibly could. >> >> If you want to play around with it, you can download the latest >> version from here: >> >> https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/ >> >> and try it on this: >> >> https://pagure.io/hello_rpkg >> >> clime >> >> > >> > >> > [1]: https://docs.fedoraproject.org/en-US/packaging-guidelines/Versioning/ >> > [2]: https://hackmd.io/kuXOPe74RfepuztBz7pBsg >> > _______________________________________________ >> > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx >> > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx >> > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx >> _______________________________________________ >> devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx >> To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx >> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ >> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines >> List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx > > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
