I found quite a big mess today, caused by an attempt to bump perl to 5.34.1 in Fedora 36: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cea638ebd4 Because some packages depend on the exact perl interpreter version, the maintainer made a buildroot override for perl: https://bodhi.fedoraproject.org/overrides/perl-5.34.1-486.fc36 and rebuilt them. Okay. The problem with this is, we have lots of perl modules. People build them all the time. And buildroot overrides apply to *everything*. So, while the buildroot override has been valid, multiple *other* perl modules have been unwittingly built against it: perl-Scalar-List-Utils: https://koji.fedoraproject.org/koji/buildinfo?buildID=1936732 perl-Parallel-Pipes: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937527 perl-PPIx-Regexp: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937522 perl-Crypt-SSLeay: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937521 perl-Crypt-OpenSSL-PKCS10: https://koji.fedoraproject.org/koji/buildinfo?buildID=1937471 ...and those are just the ones I found so far. Because they were built against 5.34.1, all of those builds require "perl(:MODULE_COMPAT_5.34.1)" , which means they require perl-5.34.1 from updates-testing. They cannot be installed with perl-5.34.0 from stable. But the maintainers likely had no idea about this - buildroot overrides are not at all discoverable, there is zero indication your package is building against one when you build it - and have created separate updates for those packages: https://bodhi.fedoraproject.org/updates/FEDORA-2022-cb1170abf2 https://bodhi.fedoraproject.org/updates/FEDORA-2022-d83d0ba901 https://bodhi.fedoraproject.org/updates/FEDORA-2022-fac98e635f https://bodhi.fedoraproject.org/updates/FEDORA-2022-c2203f1964 https://bodhi.fedoraproject.org/updates/FEDORA-2022-0990e3309e Users testing those updates will likely not notice the problem, because they'll have *all* of updates-testing enabled when they update, and perl-5.34.1 will be pulled in. But if any of those updates is pushed stable before the perl update is pushed stable, it will fail to install for anyone who does not have updates-testing enabled. This is quite a mess. I'm going to try and clean it up, but it'll be a bit ugly (there are a couple of ways I can do it, but both will involve doing bumps-and-rebuilds of the affected packages with no changes). I've been worried about this sort of thing happening for a while due to the undesirable properties of buildroot overrides. This is the biggest real-world case I've seen, but it could certainly happen again. It might even have happened without anyone noticing exactly what was going on (just mysterious dependency issues that magically went away after a bit). So: now we have convenient self-service side tags, *please use them*. Especially for something as major as a bump of perl that changes dependencies of packages built against it like this. Side tags avoid this mess entirely. Using the mechanism to produce an update from a side tag also makes it easier to make sure all the right packages are in the update and they don't get incorrectly submitted as separate updates. Thanks folks! -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net _______________________________________________ devel mailing list -- devel@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to devel-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure