On Tue, 2016-07-12 at 23:52 -0700, Adam Williamson wrote: > > Of course, we don't *have* to pick one thing or the other necessarily; > we can certainly provide all the appropriate hooks for packages to do > automated update testing, this is something folks are already looking > at, and there's no reason to stand in the way of maintainers / teams > who want to implement package-level automated tests for distribution > updates. Just to follow on from that a bit, I guess it's worth noting that the current Bodhi system just isn't a great fit for some packages; it's a bit of a one-size-fits-all model, but it fits certain packages much better than others. Bodhi's *existence* is a result of several cases where updates got pushed out that badly broke basic system functionality for a large number of users, and we got terrible press; thus it's kinda designed around avoiding *that*. If you look at the design of Bodhi 1 especially you can see how that's the problem it was trying to solve; Bodhi 2 is kinda an attempt to solve some limitations we rubbed up against in Bodhi 1 use, but it's still fundamentally an elaboration on the same basic design. So Bodhi works best for updates to familiar tools and applications, and things that are involved in basic system functionality for lots of testers. You can really see the value of Bodhi best when someone, say, pushes an update that breaks boot on encrypted systems (or something like that) and the update gets a flood of negative karma and doesn't go out. That's Bodhi doing exactly what it was designed to do, and that's what it's best at. It's almost equally good when there's, say, a trivial update to gedit, and it gets a prompt flood of positive karma, so the update goes out with almost no delay. But Bodhi kinda struggles with, well, the kind of update that started this whole discussion. fedmsg-meta-fedora-infrastructure is an extremely specialized package with an extremely specialized function that, realistically, Joe Q. Random Tester is not going to be able to test effectively. In an ideal world, Bodhi just isn't a very good fit for it...but Bodhi is all we have, so we make the best of it. I don't actually know that there's any sensible instruction we can give rookie testers for giving feedback on a fedmsg-meta-fedora-infrastructure update besides 'just leave it alone and move onto something easier'. So the system's definitely sub-optimal in this case, and it would be great to have something better. If we could provide the necessary hooks for automated testing this might actually be a good case for it, if the fedmsg maintainers were willing to develop some kind of automated test that deployed a fedmsg instance and ran some basic checks on it, which we could hook up to appropriate package updates...the biggest objection to more heavy automation of package update testing is that automation can miss problems a human would catch, but I think a very specialized package like this is actually a good example of a case where automation is more likely to do a good job than Random Human Testing. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://lists.fedoraproject.org/admin/lists/devel@xxxxxxxxxxxxxxxxxxxxxxx
