This is a conversation which has started a few times but not in anything close to a formal way. Let's fix that so we at least have an answer of why we're doing what we're doing :) A test case management system (TCMS) is pretty much what it sounds like - a system which manages at least test cases for an organization. In general, a TCMS is going to also manage test plans and many also keep track of results. Fedora doesn't have a traditional TCMS - we use the wiki to store our test cases and the test matrices which roughly align to test cases, test plans and results. The basic question is: "should we be using a more traditional TCMS instead of the wiki?". There aren't a ton of alternatives available to us, so the comparison list is pretty short. As far as I know, our options are: 1. Start using Kiwi TCMS 2. Create, deploy and maintain our own custom TCMS 3. Keep using the wiki for our TCMS If anyone is aware of other options which don't sacrifice the functionality we use for release validation, are open source and aren't abandoned upstream, please reply to this thread. Just because these 3 are the only options I could find doesn't mean that there aren't others. I've written up a brief summary of what I see of the advantages and disadvantages to the three approaches to start the conversation. What are peoples' thoughts on our options going forward? Should we look at moving on from our existing wiki-based TCMS? If so, do either of the alternatives appear promising enough to move forward with? Thanks in advance for everyone's time and thoughts, Tim
================================================================================ 1. Start using Kiwi TCMS ================================================================================ Advantages: - Has an existing user base, we wouldn't have to create/maintain an entire TCMS alone - Uses a more structured format which can be easier to analyze - Allows multiple people to report results at the same time Disadvantages: - Has strict relationships between products, test cases and test plans which don't align well to how Fedora QA currently does things - Is currently missing some vital features (FAS integration) - Would require us to deploy and maintain an instance - Would likely require some custom patches/plutins that we'd have to keep working with any changes in upstream KiwiTCMS - Does not support testcases in multiple languages KiwiTCMS (https://kiwitcms.org/) is an open source (GPL2) TCMS which started life inside Red Hat managing test cases for RHEL. As far as I know, it's the only open source TCMS out there which is still maintained and is even close to our workflow. I have set up a demo instance for people to poke at, details are in a separate email to this list. Kiwi is written in Django so that wouldn't be a huge hurdle for our currently available development skills but it is currently missing at least one feature before we could use it in production (FAS authentication). That being said, Kiwi is rather opinionated on what the relationship between testcases, test plans, builds and results should look like. We would have to make some non-trivial changes to how we do things like release validation and that's a non-trivial cost. ================================================================================ 2. Create, deploy and maintain our own custom TCMS ================================================================================ Advantages: - Could be whatever we wanted it to be - Would integrate well with existing Fedora systems Disadvantages: - Writing our own, custom TCMS would be expensive and take time - We'd have another system to maintain There's always the option of writing our own TCMS. While it'd be great to have something that uses ResultsDB out of the box and works perfectly with our workflows, that would come at a cost. The question comes down to whether we'd gain enough from a custom TCMS to justify the work we'd have to put into the system. ================================================================================ 3. Continue Using WikiTCMS ================================================================================ Advantages: - Doesn't require any changes to how we're doing things - Is very flexible - can do pretty much whatever we want to without a ton of code - Doesn't require much maintenance or custom code beyond wikitcms Disadvantages: - More difficult to analyze data contained in the wiki - Only one person can report results in a given matrix at a time - QA is the only group still using the Fedora wiki This kinda started a while back when CPE started asking about whether we still needed the wiki or not. The current Fedora wiki is a source of frustration for them due to the amount of spam and bot issues that they have to deal with. Seeing as how QA is one of the only (if not the only) groups left using the wiki, it's worth asking whether we really still need it or not. It does sound like most of CPE's concerns could be satisfied with a new wiki instance which could be locked down to only allow pages with a certain format but I don't believe that conversation has even been started with them. A lot of what the wiki has going for it is quite simply "if it ain't broke, don't fix it". Is the wiki an ideal TCMS? No. Does it serve our needs for the moment? Yes. There are so many things that we could be putting development time into, I'm not convinced that it's worth the time and effort needed to set up a new TCMS and change our processes to work with that new system just to say we're using a "proper" TCMS. _______________________________________________ test mailing list -- test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe send an email to test-leave@xxxxxxxxxxxxxxxxxxxxxxx Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/test@xxxxxxxxxxxxxxxxxxxxxxx Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
