On Wed, 2011-04-27 at 21:03 +0200, Tim Niemueller wrote: > On 27.04.2011 15:38, Kamil Paral wrote: > >> > >> - Post only errors It is common, for example, in automated build or > >> continuous integration systems to send out emails only on errors. > >> Similar goes for Unix tools, which tend to be quiet if everything > >> is ok, and only bother you with output if something is not. > >> Therefore, I propose to have AutoQA messages posted only in case > >> that there has been an error. > > > > How can you then distinguish an update for which the tests have > > passed from an update for which the tests haven't yet been executed? > > > > Moreover, currently not all updates are tested. Sometimes our tests > > simply don't work properly. Not just the updates are being tested, > > the whole AutoQA is being tested (and developed) in this whole > > effort. > > Please don't force testing, fixing, and maintaining AutoQA on the rest > of us. Well, we don't force it, but like bodhi, koji and most other key infrastructure tools, it is software ... and software unfortunately has bugs. We do our best to minimize those bugs, but we've found it very difficult to emulate a real-world bodhi+koji setup (with interesting data) in our test instance. > Integrate it such that stuff is pushed to testing only after > AutoQA has been run, or have a flag display "tests ran". Or post the > "PASSED" messages, but make Bodhi not sent messages in the case the > tests passed. Re: integration and only allowing updates to proceed to 'updates-testing' ... that's the goal. Unfortunately, as with many things, arriving at a destination involves a journey. We can't simply turn this "feature" on until we have confidence that the tests are correctly enforcing the package update policy [1]. Therefore, we are operating in a permissive mode at this time. With regards to having a flag or having bodhi not send messages in certain scenarios. Those are definitely options. We'll need to raise those ideas the bodhi folks [2] for review, since AutoQA and bodhi are separate projects. But your concerns are definitely worth following up on. [1] https://fedoraproject.org/wiki/Updates_Policy [2] https://fedorahosted.org/mailman/listinfo/bodhi > Keeping the current way will just make me (and possibly others) add > filters to throw away messages from AutoQA. Please be aware of how much > contributor time you waste by making them hope through useless (because > the tests have passed and no information is gained) mails. I realize you > want to improve things, but at the current stage its consuming the most > valuable resource we have, packager/developer time. > >> - Accumulate error messages An email is sent for every single > > > > We have that in plan, believe me. > > > >> Combined with the earlier proposal, the states for all platforms > >> should be collected by an intermediate node, and if and only if a > >> test failed on any of the platforms, one message with all status > >> messages is posted to the update. > > > > Sending Bodhi comments is just a quick way how to inform the > > maintainers. We are working on a results database with API that other > > Fedora services (Koji, Bodhi) could query and use the results as they > > seem fit. For most tests I expect it will be similar to what you > > describe. But that's future. Until that's implemented we can only > > either send comments to Bodhi or send no comments at all. > > I understand you like to have a quick and working solution for now, and > that great stuff is coming. But you cost a lot of time right now. Please > reconsider to make your development and testing time less intrusive for > others. Thanks for your feedback. We're actively working on ways to reduce the unnecessary emails generated by bodhi when posting feedback. Obviously, the goal is to work *for* maintainers, not against them. > >> On a related note: it'd be much appreciated if Bodhi would provide > >> an option to get a daily digest with all comments of all the > >> packages I'm involved with. > > > > Great idea, you can ask lmacken about that (or create ticket in its > > Trac). Or, you can filter your emails and check the relevant folder > > once a day :) > > That still means striving through many messages with lots of non-info > text. One concise email would make things much better. The point Kamil was making is that bodhi and AutoQA are different tools. If you have a good idea for bodhi, please raise that on https://fedorahosted.org/mailman/listinfo/bodhi. > >> I hope the fine folks of the AutoQA effort take these proposals > >> into account when proceeding in the development of the system and > >> help me to stop ignorance from taking over. > > > > It will take some time, but we see the deficiencies, same as you do. > > We try to improve as fast as possible. > > Please try to find a way to do this without costing as much time as atm. > There must be ways, for example have a list of packages to use it for > that packagers can opt-in and make AutoQA developers the first to use it. Opt-in support has been available for some time now. I probably could communicate this better, but alas ... https://jlaska.wordpress.com/2010/06/01/fedora-package-maintainers-want-test-results/ > > PS: We have a special mailing list for AutoQA: > > https://fedorahosted.org/mailman/listinfo/autoqa-devel > > Sorry, to keep me involved in Fedora I have to make it a reasonable > effort, joining yet another project is out of my possibilities atm. Understood. There are only so many hours in a day. Thanks, James
Attachment:
signature.asc
Description: This is a digitally signed message part
-- devel mailing list devel@xxxxxxxxxxxxxxxxxxxxxxx https://admin.fedoraproject.org/mailman/listinfo/devel
