On Sat, 2019-12-07 at 09:22 -0500, pmkellly@xxxxxxxxxxxx wrote: > I'm thinking about a project for myself and I'm curious about some things: > > Is there a tool available to automate (relval report-results)? Something > like a template with check boxes and text fields you could fill in as > you test and then do something like (relval post) and Yeah! all the > results would get posted. > > Is there a text file posted somewhere that contains all the command line > commands that get used in the course of release testing? Such might be > handy for those who might like to automate some or all of their testing. > Just trading searches and typing for grab one file then cut and paste. > > I want to experiment with automating most of my release testing. Then I > could test more often and not contend with my typos and such. So just to talk about the relval side here: no, nothing like this exists ATM (at least AFAIK). If you want to implement something like this, I wouldn't do it by scripting relval report-results, FWIW. relval is just a CLI front-end to python-wikitcms, really: https://pagure.io/fedora-qa/python-wikitcms That's where all the major heavy lifting is done. If you wanted to write some sort of batch-submission tool I think it'd make sense to just write it as another front-end to python-wikitcms - either as a separate little app, or as a new subcommand for relval, `relval batch- report` or something like it. I'd be happy to review a pull request to add a feature like that. > My idea is to connect 2 PCs together via USB. One PC would be running > Python pretending to be the keyboard on the test machine. So of course I > need a Python library that provides for keyboard emulation. I would > setup the commands so stdout and stderr go to files I could use to parse > results to a relval template. I understand that posting back to relval > presents more problems than just having a template. To bypass all of > that I have considered that I could restrict all such testing to > Rawhide. and just send a template of results to @test at the end. Though > I might need to do something other than that as I doubt everyone would > enjoy getting my frequent Rawhide test reports. > > I understand that some tests require ears and eyes; so results can't be > parsed from a file. Yes, it's sort of just duplicating what coconut > does, but at least it would be done on a different bare metal machine. > Also, I doubt that coconut does it this way; so it would be a bit of a > different environment. > > I suspect I'm lining up a lot of hours for myself. I doubt that the > Python library I need exists; so my first task would be to write the > library. The elapsed time is probably measured in years, but I have and > have had long term projects. The possibility that this would be a > foolish waste of time has occurred to me. Yes, this would be a lot of work :) To give you some context, almost all reports resulted from 'coconut' come from the openQA tests: https://openqa.fedoraproject.org/ The fedora_openqa library - https://pagure.io/fedora-qa/fedora_openqa - has configuration and logic for deciding when one or more openQA test results correspond to a passed wiki test, and reporting those results to the wiki. (It does this using python-wikitcms, of course). Currently we run all openQA tests in qemu virtual machines. However, openQA does actually have the capacity to run tests in other environments, and SUSE do this in production, we've just never used it (yet) in Fedora. For instance, one of these other 'backends' is called ikvm and it's designed for bare metal testing: it uses a feature called iKVM on Supermicro server motherboards, where the system firmware actually provides a VNC server, so this fits in great with openQA, which was initially written to interact with qemu virtual machines via VNC (this is still the most common way to use it, and how we use it). With iKVM, openQA can interact with an install running on bare metal via VNC much the same way it interacts with a VM via VNC. If we wanted to automate bare metal testing, using one of the other openQA backends might be the easiest way to do it, since we have openQA running and tests written already; the work would involve adjusting our test configuration to run some of the tests on a bare metal 'machine' using an appropriate backend, and setting up the actual hardware to do it. You could do this with a local test openQA deployment initially; to put it in production we'd need to deploy hardware in the Fedora data centre. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx
