On Thu, 17 Dec 2020 at 21:15, Fabio Valentini <decathorpe@xxxxxxxxx> wrote: > > On Thu, Dec 17, 2020 at 8:06 PM Ben Cotton <bcotton@xxxxxxxxxx> wrote: > > > > https://fedoraproject.org/wiki/Changes/Enable_Spec_File_Preprocessing > > > > > > == Summary == > > This change should enable an opt-in spec file preprocessor in Fedora > > infrastructure for the benefit of packagers. The preprocessor allows > > some very neat tricks that were impossible before, for example > > generate changelog and release automatically from git metadata or pack > > the entire dist-git repository into an rpm-source tarball (effectively > > allowing unpacked repos to live in DistGit). > > As far as I can tell, this is the third implementation of generated > changelogs ... did the autorelease / autochangelog work that was even > already deployed in staging not go anywhere? I know about it but I don't have much information about its current state. > > > == Owner == > > * Name: [[User:clime| Michal Novotný]] > > * Email: clime@xxxxxxxxxxxxxxxxx > > > > > > == Detailed Description == > > > > There is a recently added feature into mock: > > [https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor > > the rpkg preprocessor] which, if enabled, introduces an intermediate > > step just before srpm building. This step consists of running the spec > > file through a text preprocessing engine that includes an already > > present library of macros designed specifically for rpm spec file > > generation from git metadata. This library is called > > [https://docs.pagure.org/rpkg-util/v3/macro_reference.html > > rpkg-macros]. The macros there allow packagers to have their > > `%changelog`, `Release`, `Version`, `VCS` tag, or even `Source` fields > > automatically generated from dist-git repository data and metadata. > > The library can be easily extended in future to support more packager > > use-cases or even a completely new library can be developed that > > doesn't look at git metadata at all and instead, for example, analyses > > already present tarball content to render spec file based on upstream > > information. This doesn't mean it will happen but the framework is > > generic enough to support that. There is also support for user-defined > > macros that are loaded on-demand from a file placed alongside the > > package sources, maintained by packager. This feature wouldn't be > > enabled by this change from start but it's an example of freedom that > > the preprocessing framework is able to provide. Enabling this change > > should be very easy, basically adding: > > > > <pre> > > config_opts['plugin_conf']['rpkg_preprocessor_enable'] = True > > </pre> > > > > into mock configuration of Koji builders and using at least mock 2.7. > > Some very minor change may be also needed in Koji regarding the spec > > file lookup. > > > > Even if the change is enabled on the infrastructure level like this, > > the packager will still need to opt-in to use the preprocessor. The > > opting-in is done by placing `rpkg.conf` file into the package > > top-level directory with the following content: > > > > <pre> > > [rpkg] > > preprocess_spec = True > > </pre> > > > > When this is done by a packager, the preprocessor will be finally > > enabled for the given package. > > > > Alongside, there is an ongoing work to add the preprocessor support > > into the `rpkg` python library so that a packager can easily work with > > the spec files containing the preprocessor (rpkg) macros: > > https://pagure.io/rpkg/pull-request/530 > > > > The macros are supported since epel6 until the most latest Fedora > > (preproc, rpkg-macros, and rpm-git-tag-sort packages are needed). The > > spec preprocessing step in mock happens in a target chroot just before > > the srpm build. > > > > > > == Benefit to Fedora == > > > > This change offers solution to some long-standing issues in Fedora > > around packaging (i.e. automatic release and changelog generation) > > while also offering some interesting future options (for example > > unpacked dist-git repos). The big advantage of this approach is that > > it is explicit. Spec file stays the source of truth and by looking > > inside one, you will be able to determine how the text will expand for > > a certain git repository state. > > What does "unpacked dist-git repos" mean? Is this a euphemism for source-git? In this context, it means a dist-git repository with the original upstream sources placed directly in it (i.e. the sources are not packed in a tarball but they are directly present in a dist-git repo). To remind us of our collective knowledge, there can be multiple variants of this. One of them is that the spec file is placed next to the unpacked upstream sources (the simplest one). Then there are approaches that try to keep the upstream (unpacked) sources and the spec file separate - i.e. they can be in different branches, different repos, different subdirectories - each of these use-cases can be supported by the preprocessing engine. But I would say this is an advanced feature of the macros and it can be forbidden to use from the start (effectively git_dir_pack and git_dir_archive macros wouldn't be allowed). But I will be happy to discuss it. It is related to source-git although the meaning is very slightly different with respect to the separation of upstream and the respective downstream sources - in source-git, they, by default, live in completely different source forges and some synchronization between the two is needed. Here, when I say "unpacked repo" I mean a repo with mirrored upstream sources that live directly in our dist-git. > > > == Scope == > > * Proposal owners: > > For the very basic support, probably a small patch in Koji is needed > > to be able to lookup not only `.spec` files but also `.spec.rpkg` > > files (the `.spec.rpkg` extension explicitly states that the spec file > > is a template). Also the `rpmdevtools/rpmdev-bumpspec` script should > > be tweaked to be compatible with spec files using the macros. > > Is the Change owner going to submit patches for fedpkg and rpmdevtools? Yes, he will unless a rpmdevtools devel won't be faster :). > > > * Other developers: > > Some optional help with `rpmdevtools/rpmdev-bumpspec` changes would be welcome. > > rpmdev-bumpspec is a tangled mess of spaghetti code and I'd rather not > touch it or make it even more complicated. > I also think this can't be optional. releng uses rpmdev-bumpspec as > part of scripted mass rebuilds, so it must work with all packages. I meant "optional" in sense that I would optionally welcome some help here. I can probably do it on my own, i.e. open a PR and continue from there but I also feel like it would be great if this was more of a collective effort. > > > * Release engineering: [https://pagure.io/releng/issue/9910 #9910] (a > > check of an impact with Release Engineering is needed) > > Enabling the rpkg_preprocessor plugin in mock config for Koji builders. > > > > * Policies and guidelines: > > The new macro support should be mentioned or even described in the > > packaging guidelines. We should decide if the full power of the > > rpkg-macros library should be allowed from start (i.e. even unpacked > > repos). > > > > * Trademark approval: N/A (not needed for this Change) > > > > * Alignment with Objectives: N/A > > > > == Upgrade/compatibility impact == > > Because of the opt-in nature on packager side, there should be no > > compatibility issues. > > > > == How To Test == > > Once the feature is enabled, one can test it by providing the > > `rpkg.conf` file with the required content in a package repository and > > use some rpkg macro in the spec file: e.g. > > > > <pre> > > Name: {{{ git_dir_name }}} > > </pre> > > > > to generate the name of the package from the repository name (this > > should actually produce the original text as package names should be > > the same as the repository basenames). > > Not sure I understand, but what's the benefit of making the Name a > macro determined by the repository name? > As stated, they must always be the same anyway, so why make this a > dynamic templated value? There is not much value in this particular usage. I just tried to recommend a way to test it out with the least amount of effort. > > > To try it out first without committing to dist-git, one can use `rpkg` > > command-line tool from > > https://copr.fedorainfracloud.org/coprs/clime/rpkg-util/ or even > > fedpkg's koji scratch build after > > [https://pagure.io/rpkg/pull-request/530 the work in the pyrpkg] > > library is finished. > > > > One can also currently use Copr's SCM "rpkg" build method where the > > macros are enabled but the rpkg-macros there are in version 2 whereas > > this change is about introducing the > > [https://docs.pagure.org/rpkg-util/v3/macro_reference.html version 3 > > rpkg-macros]. However, while there are some differences between v2 and > > v3, the idea and most of the working is the same. > > > > == User Experience == > > This change is intended for packagers. It should help to make a bit of > > their work easier and offer them some new interesting options. > > > > == Dependencies == > > N/A > > > > == Contingency Plan == > > > > Packagers can opt-out on individual basis by removing the `rpkg.conf` > > file or just setting the `preproces_spec` property to `False`. On > > infrastructure level, the rpkg_preprocessor plugin could be disabled > > again. > > > > == Documentation == > > - [https://github.com/rpm-software-management/mock/wiki/Plugin-rpkg-preprocessor > > Mock's rpkg_preprocessor plugin] > > > > - [https://docs.pagure.org/rpkg-util/v3/macro_reference.html > > rpkg-macros reference (the library of macros ready to be used in spec > > files)] > > > > == Release Notes == > > Currently N/A > > > > > > -- > > Ben Cotton > > He / Him / His > > Senior Program Manager, Fedora & CentOS Stream > > Red Hat > > TZ=America/Indiana/Indianapolis > > _______________________________________________ > > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx > _______________________________________________ > devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx
