> We're really > reaching the point where we need to do at least ii) and iii) in a rather > more sophisticated way. > > we now have a couple of efforts that are coming at ii) and iii): Test > Maps and testcase_stats - > http://testdays.qa.fedoraproject.org/testcase_stats/ . They're both > pretty neat little hacks, I reckon, and they're both things we ought to > have. But I think in an ordered conception of things, we really ought to > be thinking about coming up with a more robust *framework* for test > plans and test results, which would allow us to build things like test > maps and testcase_stats as fairly thin 'presentation layers' on top of > the robust underlying framework. I think Adam described the basic problem really well. testcase_stats is an example of our attempt to have some useful information extracted from a wiki page full of test results. But it hits the wiki parsing limits very easily, and beyond that the reliability of this approach plummets. I like Test Maps idea and I think we can make it useful to a certain degree, but I wouldn't put too much effort into it, as long as we build it on unstable foundations (results in a wiki page). Unfortunately, all our development effort now goes to Taskotron, so I don't know what we can do about a proper TCMS solution in the near future. So, when talking about best-effort simple-hack Test Maps tools (based on wiki results), I think this could be reasonable: * Create several pre-defined test maps users can choose from on start (depending on their hw and skill set). Don't aim for too much variability, except for some simple choices in certain steps (e.g. preferred DE to install, or bios/uefi choice, or disk driver choice, etc). * Limit the number of "requirements" people can filter the test cases with. Maintaining this kind of metadata outside of test cases is going to be quite time consuming and often out of sync. Provide just the most basic ones. (To tell the truth, I don't really understand what Update Requirements button does). * Maybe display the test cases in an iframe directly from the wiki in order to avoid formatting issues (like QA:Testcase Mediakit ISO Checksums and the inline icons). * For each test map offered, display the current results for included test cases (probably per test case and not per arch, because that's really hard to parse). The parsing code can be taken from testcase_stats. Which this displayed, the testers can easily select test maps which have the smallest test coverage at that time. If we wanted anything more advanced, the wiki-based results would not be sufficient and we would have to implement something better first (on top of ResultsDB or something else), I believe. -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
