On 03/17/2017 06:23 PM, Adam Williamson wrote: > Hi folks! > > So Colin and I had a bit of a chat about openQA Atomic testing > 'integration', and one of the things that came up was the state of the > 'compose check report' emails for Atomic tests. > > If you're signed up to test@ or devel@ you may have noticed that a > 'compose check report' mail is sent out for every mainline Fedora > compose openQA tests, with info on the numbers of passed and failed > tests and some other stuff. > > When we set up openQA to test the Atomic Host OStree installer image > and to test the nightly 'two-week Atomic' composes, I asked whether we > should generate those mails, and who they should go to. > > The answer I got - and the way it's configured right now - was that > people only wanted the mails produced when a test *failed*, and they > should be mailed to this list plus to one person directly (I think at > first it was Adam Miller, it now seems to be Mike McGrath). > > I've just verified that this is actually working: when openQA tests the > 'two-week Atomic' composes and a failure occurs, the mail does get > sent. However, there hasn't been a failure of the test since - AFAICS - > 2016-10-01. So that's why no such mails have been sent out recently. I > got Mike to check, and he actually did get a mail on 20167-10-01, when > the test last failed. woot for passing tests! Adam, do you know if there is any way to send a weekly report that basically states how many tests ran and how many passed? That would at least let us know that they were there and still running and passing. I imagine now that the tests results are in resultsdb the answer is going to be something related to that. > > I think the mails that have been sent to cloud@ were never approved > through moderation, so they never actually appeared here. It'd be good > if a list moderator could check if they see a few 'compose check' mails > hung up in moderation, or something. I found this thread from a long time ago on this topic: https://lists.fedoraproject.org/archives/list/cloud@xxxxxxxxxxxxxxxxxxxxxxx/thread/OUSTJ7WHDCO5WEP7J57B6H5GYWAAF4EM/#OUSTJ7WHDCO5WEP7J57B6H5GYWAAF4EM Looks like someone tried to add that mail address to the list but doesn't seem to be working. These emails are not in the held messages from what I can tell. I think I am a moderator and not an administrator. https://wiki.list.org/DOC/Difference%20Between%20a%20Moderator%20and%20Administrator I need to get jzb to make me an admin so I can investigate. > > So, I just wanted to give a quick heads-up on that, and say that if > anyone would like that configuration changed, I can do it easily > enough. We could have a report generated for every compose as for the > 'main' composes, but it'd be quite dull I think, because there's only > one test and it almost always passes. We can also change the list of > addresses that receive the mails when they're sent, if it should be > changed. Indeed. Thanks. I think the real answer in the long run is to have dashboards that display test results in a nice way (i'm talking flashy gui web page stuff). We could have one dashboard that highlighted failures in atomic host across different composes and then a dashboard that highlighted failures/passes in a particular or for a particular ostree. I'm not asking *YOU* for this, but I think that is the answer long term. The emails would then have a link to the test results as well as the dashboard where you can see all test results. > Also do let me know if running more tests on the Atomic installer image > would be desirable. For now we just run a single test - a straight- > through install test on x86_64 BIOS - since that's all I was asked for > initially. We could run the same test on UEFI, if desired, and we could > run some of the post-install tests that are run on other images, and we > could run some of the install variant tests; I just don't know which > ones are relevant / useful for the Atomic installer image. > Yes! we would love a UEFI test and I think it would actually be good to run the atomic-host-tests against these images assuming openQA is the right tool for that job. I thought openQA was more for interactive install testing, so please let me know where I'm wrong. roshi knows openQA and atomic-host-tests so he might be able to comment here. Thanks Adam, Dusty _______________________________________________ cloud mailing list -- cloud@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to cloud-leave@xxxxxxxxxxxxxxxxxxxxxxx
