> On Wed, 2014-07-09 at 07:52 -0400, Kamil Paral wrote: > > > Hi, folks. > > > > > > So, we're scheduled for Alpha TC1 tomorrow. We had a nice happy > > > co-operative plan where QA and the WGs would collaborate on revising the > > > release validation test process for Fedora.next... > > > > > > ...which, well, didn't really happen. As of this morning we were nowhere > > > near having a viable validation process. So I went for plan B: I spent > > > today more or less pulling the entire thing out of my ass. > > > > Lots of work has been done. Thanks, Adam. > > > > When I look at all those blank cells waiting to be filled in, I'm > > getting a bit dizzy :-) > > > > Do you think it would make sense to use the same approach as we used > > in Rawhide test matrices for selected F21 matrices as well? I.e. > > instead of erasing all cells every time a new TC is released, we would > > use the "put date/TC number into the cell" approach? I think it would > > help us to improve our coverage - currently we don't have a good > > overview of which test cases were tested regularly in previous TCs and > > which were not at all, because it's hard to consolidate and compare > > the past results. For RCs we could use the traditional approach of a > > fresh new wiki page, but for TCs the new approach could be helpful, I > > think. > > > > Especially in small matrices like Server or Base it could work well, > > but even for larger ones it might be worth the try. And it would be a > > bit less depressive than to see your results "erased" with ever new > > compose. > > > > I have modified the results template a bit (the username is now shown > > as a tooltip to reduce clutter, thanks jskladan for help) and the > > result could look like this: > > > > https://fedoraproject.org/wiki/User:Kparal/Results_Template_Alternative > > > > We could set up some recommendations for when to keep the older TC > > results and when to overwrite (e.g. from TC1 pass -> TC2 pass), and/or > > maintain the matrix manually (remove old data when needed). There are > > a lot of options in this area. > > > > What do you think? > > Hah - when i was two paragraphs into your mail, I started thinking 'hmm, > we could amend the Result template' :P - great minds think alike, it > seems... > > In a way it'd be cleaner to have the image as part of the template, but > then you'd have to 'pad' the entry like this: > > {{result|pass|adamwill||||TC1}} If you think it's better to have it as a part of the template (probably is), let's change to order: {{result|pass|adamwill|TC1}} {{result|fail|adamwill|TC1|123456}} {{result|fail|adamwill|TC1|123456|111222}} I think that's no less readable than the original schema. > > which would kinda trip people up (you'd have to remember the template > has space for exactly three bug references, and pad correctly...recipe > for disaster, I guess.) Well, we could try and get clever in the > template and heuristically determine if any of the parameters after the > username is a bug ID or a build and behave accordingly, but that may be > gilding the lily. :P I haven't studied the templating markup language too much, but I doubt you can express such programming structures in there. But if we place the TC identifier before the bug numbers, as seen above, the problem should be solved. > > We *do* have http://testdays.qa.fedoraproject.org/testcase_stats/ , > which is kind of in the 'cute nonsense hack' category but does work > pretty well, to the extent that I've started using it as an entry point > for testing, sometimes. I had a quick chat with Josef about this. There is a problem with table column in our matrices. Historically we used it mainly for separating architectures and they used to be pretty similar between the tables. With F21, the column headers now look like: Server Workstation Spins ARM disks Cloud i386 x86_64 UEFI ARM i386 x86_64 EFI Ext boot VFAT boot i386 x86_64 i386 x86_64 ARM x86[1] ARM x86 BIOS[1] x86 UEFI ARM LVM ext4 NTFS BTRFS i386 x86_64 EFI ARM That requires non-trivial semantic hardcoding inside the script (for example by the section name), if we don't want to squash all columns together a produce a single number. As long it only differed by architectures, it wasn't such a problem, usually the bugs were not specific to architecture. But now there are large differences between the columns, e.g. Server vs Workstation vs Cloud, or LVM vs ext4 vs BTRFS. So, some maintenance would be needed to make the script work with F21 matrices, and either we need to hardcode a pile of hacks which would get broken by every template update, or we would need to skip certain section tables completely. That's why I'm looking into alternative approaches, which would be a bit more manageable, and rawhide-style wiki matrices seem like a viable approach to try. > > We also do have an existing mechanism and policy for transferring > results from one build to the next, but doing so is pretty clunky and > it'd definitely be easier just to use the approach we've been using on > the monthly pages. > > So...I guess I'm percolating this one, it seems decent but I just want > to think about it a bit more. What do other folks think? My only concern is that things would get out of hand and the tables would start to grow a lot. But in that case we can either prune it by hand when needed, or refresh the whole page (for example we have a single page for TC1/TC2/TC3, then there's a lot of results, so we create a fresh new page for TC4-TCx. Rinse and repeat). In this case it would still be relatively easy to spot test cases which were not tested in a long time, because you would have approximately just one historical page (going not too far into history) to check and compare to the current one, which is manageable (if the current page is for TC4-TC5, and I can see results for TC1-TC3 summarized on a single historic page, I can then easily compare blank cells). -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
