On Sat, 2013-12-14 at 10:56 -0700, Chris Murphy wrote: > On Dec 14, 2013, at 12:48 AM, Adam Williamson <awilliam@xxxxxxxxxx> wrote: > > > On Sat, 2013-12-14 at 00:06 -0700, Chris Murphy wrote: > >>> > >> In my opinion, every one of those tests requires a feature owner. If > >> no one volunteers, if a hand off isn't made, the functionality for the > >> feature represented by the sanity check shall be removed from the next > >> version of Fedora. > > > > Do you mean someone who is responsible for development of the feature, > > or testing it? > > Testing it. > > > Right now I'm simply trying to figure out a vaguely practical approach > > for testing what we can of the installer's storage functions. That's > > really all I'm shooting for. > > I understand that. I'm suggesting an approach that ties functionality > retention to community interest. If we can't recruit even temp "QA > people" to adopt a test case, then maybe the community doesn't really > value the functions indicated by those test cases. Personally I don't think that approach really works out; what are we going to do, have a big database of who 'owns' what test at any given time? You have to file paperwork to keep tests in the matrix? Who's going to own the process of tracking who owns which tests? I think tests with difficult requirements are something we have to deal with, but I don't think having 'feature owners' is the way to go, personally. > For example the iSCSI test case. It seemed pretty much no one in QA > really cared about that functionality, as they didn't depend on or use > it themselves. It was just a test case to them. So where are the > people who actually want that function to work? I was perfectly willing to test this one, actually, only it turns out my iSCSI target has some kind of weird issue that others don't. So I can't, practically, at the moment. I can see how your approach kind of feels like it makes sense if you think of every little attribute of the installer as a 'feature', but that itself doesn't quite work, for me. I mean, 'install into free space' isn't really a 'feature', or at least I don't see that it gets us anywhere to think of it as one. What does it mean to be the 'feature owner' for the 'install onto an empty disk' 'feature'? > Another example is LVM Thin P. I want to know where the feature's > owners have been this whole time, and how it is they didn't test RC1 > to see if their own feature, given prime real estate in the installer, > was working. On that basis alone I'd say LVM Thin P should be pulled > due to lack of community interest, including apparently lack of > interest by the feature's own owners. I'm honestly willing to cut them some slack here, given that RC1 existed for *one whole day* before Go/No-Go. One thing I should probably unpack explicitly here is that I'm worried that the way we've done releases the last few cycles is becoming the New Normal, especially for people who've mostly been involved with validation in the last few releases. I don't think it's a good thing at all that we've got 'accustomed' to spinning the RC we wind up releasing about 16 hours before we sign off on its release; we only really started doing that a lot around F18, and it is not at all the optimal way to do things. It is much better if we have the RC we're planning to ship built for, like, 3-4 days, to give us a chance to find issues in it that aren't immediately screamingly obvious from the validation matrices. Building the release image late on Wednesday then doing go/no-go on Thursday is absolutely not how we really _want_ to be doing things. People outside QA aren't reading the test@ list 24/7 ready to jump on a new RC, I don't think it's entirely realistic to expect every Change owner to catch the final RC in a 16-hour window and test their Change. thinp support as a feature is actually owned by dlehman - https://fedoraproject.org/wiki/Changes/InstallerLVMThinProvisioningSupport - who I expect was taking a well-earned break during the very short window we were testing RC1. > > This is one possible approach, there are > > many others. I mean, prior to newUI, we placed a _much_ lower > emphasis > > on custom partitioning. > > Yes, well you know how I feel about manual partitioning. The test > cases are necessary to qualify the function works sufficiently well > for release. But to execute the test cases requires either people or > some magical automation. anaconda team is currently working on implementing CI into their development process, and may come to us for help with that later in the process (to help write test cases). I think that will help quite a lot; we could probably do much better 'sanity checking' with automated tests to stress obvious corner cases like setting things to invalid values, repeatedly changing values and so on. I would expect we would also be able to cover a lot of the cases in that matrix at least in non-UI form. I think there are always likely to be bugs in the UI stuff that only become apparent when you run the full interactive installer and click on stuff, and I personally am fairly sceptical about the practicality of automated interactive testing, so I think we'll need to maintain some kind of set of manual interactive partitioning tests for the foreseeable future. But I'm definitely hoping this CI project works out as I think it will take a lot of the load outside of interactive-specific bugs off of us. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
