Re: Future of Test Cases and their Management in Fedora QA

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 





On 6/22/22 14:13, Adam Williamson wrote:
On Wed, 2022-06-22 at 08:57 -0600, Tim Flink wrote:


================================================================================
3. Continue Using WikiTCMS
================================================================================

Advantages:
   - Doesn't require any changes to how we're doing things
   - Is very flexible - can do pretty much whatever we want to without a ton of code
   - Doesn't require much maintenance or custom code beyond wikitcms

Another advantage of the current system that I'd like to highlight is
that it allows us to synthesize manual and automated test results. That
is, in the current wiki results pages, you see the results from openQA
and human testers together.

I would definitely want any replacement for it to preserve this
advantage - i.e., I would want it to be able to track results from both
human testers and automated systems. Ideally, this would be achieved by
resultsdb integration - it's a bit difficult to imagine technically how
this could work, but it would be nice for results from both human
testers and automated systems to be in resultsdb, and for any human-
readable representation of the validation state to pull both sets of
results from resultsdb.

For what it's worth, Kiwi does have a pretty comprehensive API when it comes to stuff like reading or posting test results.

When it comes to storing results for automated vs. manual testing, Kiwi does seem to focus more on having test cases which are marked as automated rather than having users marked as bots. As far as I know, there is no way to mark a Kiwi user as a bot.

Personally, I think there are arguments for both approaches but changing over to Kiwi would likely require a change to how we've been doing things.

wikitcms gives us the integration of the two types of result, but it
does not currently forward results from human testers to resultsdb. So
we only have the results from automated systems in resultsdb. openQA
reports its results to *both* resultsdb and the wiki, so the wiki is
the only canonical store of *both* human and bot results.

I've toyed in the past with the idea of having `relval report-results`
also submit the result to resultsdb, but it requires solving
authentication and we still wouldn't capture human test results
submitted by editing the wiki directly. Another option would be a
wikitcms-based bot which monitored result pages for edits and forwarded
any 'new results' to resultsdb...

At some point, we'd end up with enough chewing gum and bailing wire that we might as well roll our own. I'm not sure we're there yet but if we start down the path of an alternate interface to post results and/or a bot to shuffle results between the wiki and resultsdb, I do think that would start approaching the point where we'd be better off writing our own solution.

Disadvantages:
   - More difficult to analyze data contained in the wiki
   - Only one person can report results in a given matrix at a time
   - QA is the only group still using the Fedora wiki

This kinda started a while back when CPE started asking about whether
we still needed the wiki or not. The current Fedora wiki is a source
of frustration for them due to the amount of spam and bot issues that
they have to deal with. Seeing as how QA is one of the only (if not
the only) groups left using the wiki, it's worth asking whether we
really still need it or not. It does sound like most of CPE's
concerns could be satisfied with a new wiki instance which could be
locked down to only allow pages with a certain format but I don't
believe that conversation has even been started with them.

If they do get to the point where they really, really want to turn the
wiki off, I'm willing to try and handle this (setting up a reduced wiki
instance specifically for this purpose, with restrictions to allowable
page names and so on).
_______________________________________________
test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure




[Index of Archives]     [Fedora Desktop]     [Fedora SELinux]     [Photo Sharing]     [Yosemite Forum]     [KDE Users]

  Powered by Linux