On Thu, 2011-01-20 at 14:03 -0500, Samuel Greenfeld wrote: +AD4 A few more inline comments on a subset of the questions, and two more thoughts: +AD4 +AD4 On Thu, Jan 20, 2011 at 11:54 AM, James Laska +ADw-jlaska+AEA-redhat.com+AD4 wrote: +AD4 +AD4 On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote: +AD4 +AD4 +AD4 1. What is the history of Nitrate and the Fedora Project? What +AD4 +AD4 +AD4 does the Fedora project expect to gain from using it? +AD4 +AD4 +AD4 +AD4 This goes back to an eval we did using testopia in Fedora many releases +AD4 +AD4 agove. Unfortunately, the effort was canceled due to license +AD4 +AD4 incompatibility between Fedora and testopia. At that point, we invested +AD4 +AD4 in leveraging the wiki to best of our ability. +AD4 +AD4 +AD4 +AD4 https://fedoraproject.org/wiki/QA/Testopia+-AF8-Evaluation +AD4 +AD4 https://bugzilla.redhat.com/show+AF8-bug.cgi?id+AD0-450013 +AD4 +AD4 +AD4-From the bug I'm guessing this is because Testopia used Ext-JS, and +AD4 Ext-JS kept changing licenses. Hopefully there will only be licensing +AD4 restrictions for Nitrate on the software itself, and not the created +AD4 work of what one does while actively using it. That was a question I had opened with Hurry. I have no idea how we'd license the content in the system. If anything, I'd hope/expect that to match that of our current wiki. +AD4 +AD4 +AD4 1. How does Nitrate compare to other open +ACY closed sourced TCMS +AD4 +AD4 +AD4 solutions? Why was it written as opposed to using an existing +AD4 +AD4 +AD4 solution, and what are its strengths +ACY weaknesses? +AD4 +AD4 +AD4 +AD4 See history comments above. Also, maybe the nitrate developers +AFs-1+AF0 can +AD4 +AD4 offer more insight on how it compares to other open-source solutions? I +AD4 +AD4 +ACo-think+ACo that comparison work has been done in the past, I'm just not +AD4 +AD4 sure where to find the results. +AD4 +AD4 +AD4 +AD4 +AFs-1+AF0 https://fedorahosted.org/mailman/listinfo/nitrate-devel +AD4 +AD4 Are the developers actively monitoring/using this list? I looked at +AD4 it before, and all I saw were three test posts from July in the +AD4 archives. I believe they do, but don't yet have a strong upstream presence since there hasn't been a lot of code/progress to share until recently. This will be something I'm sure they'll want to improve as community interest increases. +AD4 +AD4 +AD4 1. Can multiple projects share test cases, and even reference +AD4 +AD4 +AD4 older versions of test cases if they are lagging behind the +AD4 +AD4 +AD4 current rawhide/Fedora release? Will Spins be able to make +AD4 +AD4 +AD4 their own (simultaneously running) test plans? +AD4 +AD4 +AD4 +AD4 This is the hope. It's not really useful if we can only use it for +AD4 +AD4 release validation. I don't think we've fully explored how best to +AD4 +AD4 model other spins/projects, but I don't foresee big problems there. +AD4 +AD4 That will be fun to explore on the sandbox/staging instance. +AD4 +AD4 +AD4 +AD4 With regards to referencing older versions of a test case, I believe +AD4 +AD4 that support is there, although I'm not certain it's right for our +AD4 +AD4 needs. Keeping test documentation (plans and cases) updated is a pretty +AD4 +AD4 sizable maintenance challenge. I've seen many instances where support +AD4 +AD4 for versioned test cases allows test plans to suffer over time as they +AD4 +AD4 were linked against old and inaccurate test cases. +AD4 +AD4 +AD4 +AD4 Much like how the wiki is used now, we have support for linking against +AD4 +AD4 older versions of tests (wiki history), but we rarely ever use that +AD4 +AD4 feature. I expect that trend would continue in the short-term. +AD4 +AD4 The reason I bring this up is because OLPC's Spin releases tend to lag +AD4 behind the official Fedora release. For instance, we just released +AD4 our hopefully final Fedora 11-based release this past month, and are +AD4 in the early stages of the Fedora 14-based one. At least some Fedora +AD4 ARM development work may still be going on with Fedora 13 as well. +AD4 +AD4 While decently written test cases will survive somewhat over time, +AD4 major changes in GUI look +ACY feel or other areas can break their +AD4 backwards compatibility. +AD4 +AD4 Ideally Nitrate will default to using the current version of a +AD4 testcase when making a new plan, preferably following the updates of +AD4 said testcase until a result is committed which forces the test case +AD4 version to be needed. I hope so, yes+ACE :) I'm fairly certain support exists for linking against versioned cases, that is, I know it was present in testopia. Hopefully Hurry, or the nitrate folks can help here. Sounds like we need to add this to our list of requirements. +AD4 +AD4 +AD4 1. How long will historical test case results be made available? +AD4 +AD4 +AD4 +AD4 I suspect the limiting factor here is database size. I'm not aware of +AD4 +AD4 any rules or process that would require removal/archival of old results. +AD4 +AD4 However, at some point that could certainly be an issue we'd need to +AD4 +AD4 plan for. +AD4 +AD4 Again+ADs this would be to help lagging Spins and similar. Right now it +AD4 looks like Bodhi at least publicly hides information for Fedora +AD4 versions which have reached end-of-life. (Either that, or I don't +AD4 know how to find it.) That or garbage collected. Sounds like we'll need to do some further research on this front. I personally see no reason to purge results for EOL'd releases. I've often referenced old test results to understand when the last time something was tested. Test results aren't disk-hogs like builds (+ACo-cough+ACo oo.org +ACo-cough+ACo) can be, so I don't expect this to be as much of an issue. Nonetheless, you're point is valid. +AD4 I presume that Nitrate has already been determined to be scalable+ADs +AD4 otherwise it is a big risk to incorporate it into Fedora. I'll let the nitrate folks speak to scalability issues. That's definitely something we'll want to explore during a pilot. Remember too, I don't expect we'll throw away the wiki, and switch to nitrate without a good bake-in period to identify bugs/gaps etc... +AD4 Also: It might be useful to add an +ACI-unclear+ACI testcase result, similar +AD4 to how Mozilla's Litmus system does it (https://litmus.mozilla.org). I do like litmus+ACE It's a nice evolution from testopia for upstream mozilla. We don't currently have an 'unclear' test result. I'm not opposed to it, but would need better understand how that field is used, and the process around it, in litmus. Thanks, James
Attachment:
signature.asc
Description: This is a digitally signed message part
-- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
