On Mon, Feb 14, 2022 at 5:15 AM Thorsten Leemhuis <fedora@xxxxxxxxxxxxx> wrote: > > On 12.02.22 06:21, Neal Gompa wrote: > > On Fri, Feb 11, 2022 at 11:25 AM Justin Forbes <jforbes@xxxxxxxxxx> wrote: > >> On Fri, Feb 11, 2022 at 8:07 AM Thorsten Leemhuis <fedora@xxxxxxxxxxxxx> wrote: > >>> On 08.02.22 20:24, Donald Zickus wrote: > >>>> > >>>> It has been awhile since we changed how this mailing list is used. As > >>>> folks have noticed, we have increased traffic significantly over the > >>>> past couple of years to reflect the activity Red Hat developers are > >>>> performing on the Fedora kernel. > >>>> > >>>> My question to this list is around the thoughts of this activity: > >>>> * Is there too much noise? Should we throttle back? > >>>> * is the volume ok? Folks have good filters? > >>>> * Other suggestions on how we use this list? > >>>> > >>>> Trying to continue to make this mailing list useful. > >>>> > >>>> Thanks for any feedback! > >>> > >>> The mails are not a big problem for me, but they are related to a bigger > >>> question that again and again pops up in my head: > >>> > >>> Is it really a good idea to have all those totally RHEL specific patches > >>> in the Fedora rpm (and thus discussed here)? And does that approach mean > >>> that Red Hat is special, or are willing to include downstream patches > >>> for other distros as well? Say Amazon Linux? How is this handled in > >>> other packages? Or is it an issue that only happens with the kernel? > >> > >> It is a trade off, and I think one that tends to be worthwhile. The > >> patches that are included and discussed here also typically include > >> special casing with #ifdef CONFIG_RHEL_DIFFERENCES and the code path > >> does not actually impact Fedora kernel users. More importantly, those > >> patches are backed out on fedora stable branches, so they only exist > >> in Rawhide. > > Yeah, I'm glad that they are not in stable, thx for this. Still I'm not > really happy with the overall situation, as those patch increases the > patch count by quite a bit, which for me doesn't feel really in line > with our "we are close to upstream" claim -- and makes us look bad if > kernel developers, journalists and others count patches (I did that a > few times when I still wrote for a living -- and actually hope to write > an article about this sooner or later which takes a close look at how > various vendors patch their kernels). Patch count is really a marketing thing, that I am not particularly worried about. I am much more worried that the kernel we are building and running for Fedora behaves as similarly to upstream as possible. This is why having things like CONFIG_RHEL_DIFFERENCES does introduce additional code, but I think is an overall win because the Fedora kernel should be avoiding those codepaths all together. I can get problematic from the "apply this patch" to test a bug stance upstream, but given the locations for these, I expect actual conflict to be rare, and easy to fixup when it exists. > I'm not a git master, but would it maybe be possible to have those > patches in a separate branch and a separate %patch only applied for ELN? > Just wondering, as upstream development mostly happens with separate > branches, that's why I wonder why this can't be done here. But I fear > that things are slightly different in this case and might makes things > impossible in this case hard. Anyway: > This is possible, but it turns out to be a bit more of a nightmare to manage. The way it would be set up would be master is still the linus branch, a fedora-rawhide type branch would have to exist which merges in master regularly, and os-build would add the RHEL patches. Changing the spec to keep a linus tarball and separate patch sets for fedora and rhel, only applying the RHEL patch set if not fedora is trivial. What gets weird is from a maintainer or developer point of view. I would have to remember which branch to target my changes and it can get all very confusing, with different changes requiring different targets. > >> By including these and the process that brings them in to > >> begin with, we also get a much larger group of the RHEL kernel > >> developers, QA, etc paying attention to upstream and fixing issues > >> before they make it to a stable release. > > > >>> Sure, Fedora is mainly sponsored by Red Hat and all Fedora kernel > >>> developers are working for Red Hat, but somehow it feels to me like a > >>> boundary was overstepped. > >> > >> I can see that point of view, and I do understand it. There is a > >> tradeoff involved. While it does mean some additional overhead in > >> dealing with patches that do not impact Fedora, in return we do get > >> some additional kernel dev attention when we run into real issues. > > Yeah, that's good, I noticed. > > >> I > >> have been a Fedora kernel maintainer for a rather long time at this > >> point, and I can tell you that it used to be easier to get help with > >> issues from outside resources because everyone internally was busy and > >> the Fedora tree was not included in their responsibility list. This > >> has gotten considerably better since we moved to the ark process, > >> because suddenly the ark tree is on people's responsibility list. > > Sounds really good. > > > On the other hand, the workflow changes have certainly made the Fedora > > kernel more difficult for outsiders to work with. > > I think what Neal wrote here hits the nail on the head. Those RHEL > specific patches and the mails are slightly annoying, but I'd guess I'd > be more willing to accept them if the ARK process would get more fine > tuning to make things easier again for outsiders (I had hoped to > contribute patches to help a bit with that, but never found the time :-/ ). I thought this had improved somewhat. and should be better. I am open to even more improvements as well. Right now, doing things like a stable rc build is fairly trivial provided you have the git foo to maintain various remotes. I keep a local branch which is a clone of fedora-5.1x and keep a local checkout of greg's stable rc repository. Because my fedora-5.16-rc branch has a Makefile.variables which points to the linux-5.16.y-rc branch as upstream, merging it in just works as expected. A lot can be done with Makefile.variables to make a custom branch behave as expected. > My problem: with SRPMs it's easy to do this for thousands of packages in > Fedora without known anything about the particular package: > > - verify the sources are pristine > - update to a newer version > - add or remove patches > > Kernel-ark broke all that from where I stand: > > - the sources included in kernel.spec don't match upstream and it's not > obvious from reading the spec file how to verify them The source matches upstream in that they are a tarball made from linus' branch (and nothing else). While we did previously download and verify the signature of a tarball from kernel.org for actual releases, the rc patches and such were generated from git before as well. As kernel.org does use signatures for releases, you can verify those signatures in our clone just the same. Using git log --show-signature' will show you the signatures on commits if they exist. > - if you want to manually update stable kernel.spec from say 5.16.5 to > 5.16.6 you now have to modify the version number in iirc *four* > different places; this like that would never pass a package review in > Fedora. And even if you did that: then you still have to deal with the > kabi stuff, which is not of interest for Fedora either iirc. This is somewhere that definitely needs improvement, but it has not made it up on my priority list. More importantly, changing a release should not change the sources required. > - building a near vanilla kernel that still applied patches needed to > get things things building (like the gcc12 patches in rawhide recently) > used to be pretty easy, now it's hard (as can be seen by the mailing > list post about this a few days ago; I had trouble as well, but was able > to deal with them myself) This is an issue we are working to address. Right now, if things are building in rawhide, and not for you, the best bet is to look for merge requests labeled "Include in releases" The hope is to get these types of patches merged quickly now, though in the gcc12 case, there was some discussion upstream and both of those patches have gone through revisions, one as recently as yesterday. https://gitlab.com/cki-project/kernel-ark/-/merge_requests?scope=all&state=opened&label_name[]=Include%20in%20Releases > There are likely other aspects that makes things hard for Neal. I guess > with some more fine tuning things likely could be fixed or at least > improved a lot. > > > I've somewhat > > adapted to it, but I'm still not a fan of the process for the Fedora > > kernel. I also don't like the implication that RHEL's needs are more > > special in the Fedora kernel sources than any other downstream. > > This is the trade off, and I am okay with it. We have worked through some guidelines to make sure that RHEL specific downstream patches do not impact fedora kernel-builds. And there have been times when we just said no because the patch in question was not able to be isolated, and offered no value to Fedora. I do also remove the RHEL specific patches on Fedora rebases, so the stable branches do not include the patches. While the RHEL configs and files are still included in the Fedora stable branches, the only thing stopping me from removing them is having the time to do so. We already have some framework for it with "INCLUDE_RHEL_FILES" in Makefile.variables. It would just take some additional work to hide things like kabi sources and such behind that. > > And with the configs for kernel-ark, we call the RHEL configs as "ark" > > instead of "rhel", which I think is weird and we should call it what > > it is. There has been some discussion about this. It isn't that it was some trick to try to hide things. ARK has been used internally for quite some time, and has gone through many iterations before what we have now. The name doesn't particularly make sense anymore in the current context, but there is a bit of work required to change it. > And details like this IMHO really matter. For me they contribute to a > slight "RH took over control and we can't do anything about this" > feeling which makes me dislike the whole ARK approach, while I in > reality like the idea as whole (especially if the process would be used > for other packages as well). > > > Admittedly, I appreciate being able to adapt config differences > > for them when working with ARK. > > > > There's a lot of other uncomfortable thoughts and questions that come > > up when I think more about the Fedora kernel wrt workflow and > > maintainer restrictions. For example, is it even possible anymore for > > non-RH Fedora kernel maintainers? Was it ever really possible? These > > are the kind of awkward questions that I don't know if I am going to > > like the answer I get if I ask. It is just as possible as it has ever been to have non-RH Fedora kernel maintainers, which is to say, not easy, but not impossible. It is a bit easier for a non-RH kernel maintainer to contribute though. Anyone can submit an MR, and MRs to fedora-5.x branches or to Fedora specific files (configs, filters, etc) in os-build bypass a lot of the RHEL process. The Fedora kernel is still the Fedora kernel. I am 100% dedicated to the Fedora kernel. While I do tend to make some changes to fix RHEL things that will not build, it is about as far as I go. Just as I don't really work with the RHEL infrastructure at all, most RHEL developers don't work with Fedora infrastructure, so I tend to be launching the eln builds. There are people from the RHEL side who track down acks, and manage the merge workflow for RHEL specific things. I can (and do occasionally) nack things that will end up in RHEL anyway, if they have a negative impact on the Fedora kernel. I believe there is an advantage to having someone dedicated to this, who is only concerned with what is best for Fedora, and I am that person. Once it leaves kernel-ark for CentOS stream or RHEL, I am not involved at all with that tree anymore. > > If we're going to have RHEL configs in kernel-ark (remember, "ark" is > > really "rhel"), would other downstream configs be allowed? Can the > > testing apparatus be extended to build and test them? How far can we > > go here? > > Every set of configs is another list of things that take time to process, require care and feeding to maintain, and must be reviewed and set. I don't see any value in adding too many additional downstream configs. The mechanism already exists with config-overrides to maintain separate configs in a branch, and it is not particularly difficult to add another set of configs without overrides. I am looking to drop configs (i686, armv7), not add more. > > I have these questions as the CentOS Hyperscale SIG kernel maintainer > > and as a general Fedora community member. > > > >> While there is a defined workflow, there are improvements being made > >> over time, and we are still looking at places the current process can > >> be improved. This is why the thread was started. Is the mailing list > >> interaction where it should be from an automation point of view? Are > >> we sending too much? Not enough? > > Not sure. I for one missed that you added patches to make gcc12 builds > work. Did I miss them in the noise (then some tags might help to > distinguish RHEL and Fedora specific stuff -- or do we have those?) or > are those patches not mentioned on the list when they get applied (I > would like that, but maybe that's just me)? There should be no fedora specific patches, the GCC bits apply to both. All gcc patches were delivered to this list at the time that the MRs were opened. Also, the "include in releases" tag is handy for figuring out what might fix a build. The only patches that get labeled as "include in releases" are those which we can't build without. > > If I felt like the emails gave us something to do with it, I think I > > would be okay with the emails. But I don't know what I'm supposed to > > do with any of this. > > Yeah, sometimes they are kinda interesting, but at the same time I have > a feeling that I lost track of what Fedora is doing due to the either > the ARK process or all those mails reg RHEL specific things. > Most of what Fedora is doing gets handled on upstream lists, as the primary concern is getting them fixed upstream. MRs do get opened here, but typically not a whole lot of discussion around them. The upstream first focus of Fedora hasn't changed here, and will not. Justin > Ciao, Thorsten > _______________________________________________ > kernel mailing list -- kernel@xxxxxxxxxxxxxxxxxxxxxxx > To unsubscribe send an email to kernel-leave@xxxxxxxxxxxxxxxxxxxxxxx > Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: https://lists.fedoraproject.org/archives/list/kernel@xxxxxxxxxxxxxxxxxxxxxxx > Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure _______________________________________________ kernel mailing list -- kernel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to kernel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
