On Mon, 2014-07-28 at 09:16 -0400, Kamil Paral wrote:
> > Hah - when i was two paragraphs into your mail, I started thinking 'hmm,
> > we could amend the Result template' :P - great minds think alike, it
> > seems...
> >
> > In a way it'd be cleaner to have the image as part of the template, but
> > then you'd have to 'pad' the entry like this:
> >
> > {{result|pass|adamwill||||TC1}}
>
> If you think it's better to have it as a part of the template (probably is), let's change to order:
>
> {{result|pass|adamwill|TC1}}
> {{result|fail|adamwill|TC1|123456}}
> {{result|fail|adamwill|TC1|123456|111222}}
>
> I think that's no less readable than the original schema.
It isn't at all, but it has the rather large problem that it breaks the
API :) All existing {{result}} entries in all pages will be invalid.
That seems like a deal-breaker. We're kind of locked forever into the
first five fields being 'result', 'username', 'bug#', 'bug#', 'bug#', I
think, to preserve existing results.
> >
> > which would kinda trip people up (you'd have to remember the template
> > has space for exactly three bug references, and pad correctly...recipe
> > for disaster, I guess.) Well, we could try and get clever in the
> > template and heuristically determine if any of the parameters after the
> > username is a bug ID or a build and behave accordingly, but that may be
> > gilding the lily. :P
>
> I haven't studied the templating markup language too much, but I doubt
> you can express such programming structures in there.
I had a quick look and I suspect you can, but it'd get really pretty
messy.
> But if we place the TC identifier before the bug numbers, as seen
> above, the problem should be solved.
>
> >
> > We *do* have http://testdays.qa.fedoraproject.org/testcase_stats/ ,
> > which is kind of in the 'cute nonsense hack' category but does work
> > pretty well, to the extent that I've started using it as an entry point
> > for testing, sometimes.
>
> I had a quick chat with Josef about this. There is a problem with
> table column in our matrices. Historically we used it mainly for
> separating architectures and they used to be pretty similar between
> the tables. With F21, the column headers now look like:
> Server Workstation Spins ARM disks Cloud
> i386 x86_64 UEFI ARM
> i386 x86_64 EFI
> Ext boot VFAT boot
> i386 x86_64
> i386 x86_64 ARM
> x86[1] ARM
> x86 BIOS[1] x86 UEFI ARM
> LVM ext4 NTFS BTRFS
> i386 x86_64 EFI ARM
>
> That requires non-trivial semantic hardcoding inside the script (for
> example by the section name), if we don't want to squash all columns
> together a produce a single number. As long it only differed by
> architectures, it wasn't such a problem, usually the bugs were not
> specific to architecture. But now there are large differences between
> the columns, e.g. Server vs Workstation vs Cloud, or LVM vs ext4 vs
> BTRFS.
>
> So, some maintenance would be needed to make the script work with F21
> matrices, and either we need to hardcode a pile of hacks which would
> get broken by every template update, or we would need to skip certain
> section tables completely. That's why I'm looking into alternative
> approaches, which would be a bit more manageable, and rawhide-style
> wiki matrices seem like a viable approach to try.
Yeah, I can see the thought process there.
> >
> > We also do have an existing mechanism and policy for transferring
> > results from one build to the next, but doing so is pretty clunky and
> > it'd definitely be easier just to use the approach we've been using on
> > the monthly pages.
> >
> > So...I guess I'm percolating this one, it seems decent but I just want
> > to think about it a bit more. What do other folks think?
>
> My only concern is that things would get out of hand and the tables
> would start to grow a lot. But in that case we can either prune it by
> hand when needed, or refresh the whole page (for example we have a
> single page for TC1/TC2/TC3, then there's a lot of results, so we
> create a fresh new page for TC4-TCx. Rinse and repeat). In this case
> it would still be relatively easy to spot test cases which were not
> tested in a long time, because you would have approximately just one
> historical page (going not too far into history) to check and compare
> to the current one, which is manageable (if the current page is for
> TC4-TC5, and I can see results for TC1-TC3 summarized on a single
> historic page, I can then easily compare blank cells).
I guess another problem here is it makes the process somewhat more
complex; right now we have a fairly simple SOP for doing a test release
event, which is 'create the results pages and send out the mails'. I
guess it becomes 'decide whether to create a new results page, if so
name it according to this new naming scheme'. Not that bad, I guess, but
a bit more work. How did you envisage naming the pages?
(it also means you can't reliably find the correct results page by just
plugging the appropriate fields into the known format, I guess, unless
we're careful about setting up redirects.)
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
--
test mailing list
test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe:
https://admin.fedoraproject.org/mailman/listinfo/test
