With the ARM moving to primary and essentially cloud achieving the same status (from a QA perspectice), we need a better way to track and display test results for each release. The current method (1) is a wiki page with a set of columns for each architecture and rows for test cases grouped by test case type. While this is easily expandable, it quickly becomes a less than optimal method to display and manage data. What Adam and I were discussing is a web app to do this. The problem is that neither of us are UI designers. We have some ideas as to how we want information displayed, queried, etc, but that's it. Here are the relevant irc logs. 16:08 < handsome_pirate> adamw: Also, it's probably too late for F20, but I've been thinking about the huge number of columns in the test matrices issue 16:08 < handsome_pirate> adamw: Namely, I've been kicking around doing a web app 16:09 <@dgilmore> adamw: generally pxe with kickstart, but pxe and interactive works also 16:09 < adamw> handsome_pirate: yeah, too late for f20, but i'd love a more capable replacement 16:09 < adamw> handsome_pirate: i have a plan for f20 that's much simpler: table the other way around 16:09 < handsome_pirate> Indeed 16:10 < adamw> so for this test case, it'll be a table on the install matrix page but with the *test case* as the columns and the platforms as the rows 16:10 < handsome_pirate> adamw: I'll see if I can get my programming instructor to let me do it for class credit 16:10 < adamw> probably have multiple columns representing the same test case with different images 16:10 < adamw> that was my next thing to do 16:10 < adamw> handsome_pirate: it might turn into kind of a big project, note 16:11 < handsome_pirate> adamw: Indeed 16:11 < adamw> handsome_pirate: if we're going to do something better then we'd want to do something awesome, that hooks into the blocker bug webapp etc 16:11 < adamw> but don't let me project stop energy at you :P 16:11 < handsome_pirate> adamw: But, I'm not a GSoC person; I won't be going anywhere 16:11 < handsome_pirate> adamw: That would be wicked cool 16:11 < handsome_pirate> adamw: But, for starters, I want to follow KISS 16:12 < handsome_pirate> adamw: I'll ping mizmo or emichan to see about making it pretty 16:13 * handsome_pirate noms on steak and bacon tacos 16:15 < adamw> handsome_pirate: yeah, in fact, thinking about it, the matrices are the most easily drop-in-replacable thing 16:15 < adamw> you could just do a webapp which represents that layout more flexibly and that would drop right in and make things better 16:15 < adamw> good thinking 16:16 < handsome_pirate> Exactly what I was thinking to start out 16:17 < adamw> cool 16:17 < adamw> let's see, if i wrote a five minute spec for you... 16:17 < handsome_pirate> Aye 16:17 < adamw> we'd need a concept of tests, configurations, and releases 16:17 < adamw> and then a nice interface for querying the database based on those concepts 16:18 -!- streeter [streeter@nat/redhat/x-asubidnuwbirqiyg] has quit [Quit: Leaving] 16:18 < adamw> oh, and test level of course...and that might need to vary depending on the other concepts...hm 16:19 < adamw> handsome_pirate: so we need to express stuff like 'this test needs to be run on this config at alpha, on this config at beta, can optionally be run on this config, and is not applicable to this other one' 16:19 < adamw> and then some sort of awesome query engine so we can generate all sorts of different views, for working from and for searching 16:19 < adamw> SOUNDS EASY RITE?! 16:19 < handsome_pirate> adamw: For db query, maybe have some way to select a test case and then it shows a page with current results for that test case for all images 16:19 < handsome_pirate> heh 16:20 < adamw> oh and we need test category too 16:20 < handsome_pirate> Indeed 16:20 -!- tempus_fol [~tempus@gateway/tor-sasl/tempusfol] has joined #fedora-qa 16:20 < Cerlyn> Just don't try and do it within Mediawiki's own internal db unless they've made it so you don't have to flush every page to see an update 16:20 < adamw> sure, but i'm thinking stuff like 'show me all the required deployments tests for ARM that we haven't done since RC1' 16:20 < handsome_pirate> I've got something sort of floating around in my head 16:20 < adamw> Cerlyn: handsome's thinking of writing a webapp for it 16:20 < handsome_pirate> adamw: Aye 16:21 < Cerlyn> Yet another test case manager :/ 16:21 < handsome_pirate> Cerlyn: Probably use something like sqllite 16:21 < handsome_pirate> Cerlyn: Do you think there's one out there we can adopt? 16:21 -!- weld [~weld@xxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has joined #fedora-qa 16:22 < Cerlyn> handsome_pirate: I'd have to look. At one point Fedora was going to move to what RH was moving to (Nitrate) but that seem to have stopped 16:22 < handsome_pirate> adamw: Honestly, the hardest part for me is going to be the UI 16:22 < adamw> handsome_pirate: we're growing more and more test cases and test case sets and platforms, the wiki approach is definitely getting unwieldy 16:22 < Cerlyn> Mozilla was working on one which was supposed to be OSS & public friendly 16:22 < handsome_pirate> adamw: Indeed 16:22 < adamw> handsome_pirate: actually the more i think about it the more this would be awesome and we need it NOW 16:22 < handsome_pirate> adamw: Okay, I'll set to work 16:23 < adamw> Cerlyn: the neat thing here is we're just looking at a UI for the *matrices* 16:23 < adamw> Cerlyn: we're doing an end run around the whole tcms problem 16:23 < adamw> Cerlyn: the test cases can still be dumb wiki pages; what we really could do with is a much more flexible way of querying test case sets (and entering results) 16:24 < handsome_pirate> Cerlyn: All the test case managers I've seen won't do what we need to do 16:24 < Cerlyn> So your TCMS will import the Wiki data as a source 16:24 < adamw> Cerlyn: it doesn't need any data as a source. it's not a tcms. 16:24 < adamw> it's a webapp for representing views on the set of test cases. 16:24 < adamw> all it needs to know is the URLs of the test case pages. it does not care a jot about the contents. 16:25 < Cerlyn> then how does it know which test cases haven't been run yet? 16:25 < adamw> Cerlyn: we could build a simple one which just lets us look at groups of test cases and even that would be interesting, but yeah, we need it to store that 16:26 < adamw> handsome_pirate: i've been using trac's query interface a lot the last few days and it has its good points. i'm kinda thinking of something like that - a nice quick GUI 'query builder' which lets you generate all kinds of different views and save useful ones. we'd probably have a UI with some kind of overview for the current test build as the default, a set of canned queries, and the ui for building a custom one 16:27 -!- tyll [~till@fedora/tyll] has quit [Ping timeout: 246 seconds] 16:27 < adamw> Cerlyn: it shouldn't be _too_ hard to store that info, though. it really just needs to store...let's see, user foo ran test bar on configuration moo for release baa on (date) 16:28 < adamw> with result X, associated bug reports Y, freeform comment Z 16:28 < adamw> did I miss anything? 16:28 < handsome_pirate> adamw: I'm thinking that for the main overview (ie, where you could input results, get a quick view, etc), have a list of current test cases 16:29 < handsome_pirate> Click on one, and it expands to show current results and a place to input results 16:29 < handsome_pirate> Does that make sense? 16:29 < handsome_pirate> I'll follow trac's query page for setting up my own 16:29 < adamw> you might be best asking design team, like you said :) but let me think 16:29 < adamw> if we put ALL the test cases in one big list that's gonna be a bit overwhelming... 16:30 < handsome_pirate> True 16:30 * adamw tries to think what a UI would look like from a completely white page and is utterly terrified 16:30 * adamw never wants to be a designer 16:31 < handsome_pirate> I think the click on test case, it expands with info (at the least, link to wiki), test results, and place to enter test results may be a starting point 16:31 < handsome_pirate> I'll write up an email for the design list 16:31 < adamw> sure, if you just want to have a placeholder UI while you work on it it sounds fine 16:31 -!- Cerlyn [~cerlyn@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx] has quit [Remote host closed the connection] 16:32 < adamw> i suppose if you query the list in a certain way and then hit 'enter result' it should pre-fill the result thingy with your query restrictions 16:32 < adamw> (so if you query for ARM tests for F20 Alpha RC2, it should give you a report form with that config and release selected...) 16:32 < adamw> zoiks, i need to get back to working on what we have now, hehe :) i'll leave you to it At this point, any help with figuring out how to make it look would be much appreciated. Thanks much, John. (1): Example: https://fedoraproject.org/wiki/Test_Results:Fedora_19_Alpha_RC1_Install _______________________________________________ design-team mailing list design-team@xxxxxxxxxxxxxxxxxxxxxxx https://lists.fedoraproject.org/mailman/listinfo/design-team
