On Wed, 2012-10-10 at 12:17 -0400, David Cantrell wrote: > You're not understanding what I was pointing out. The blocker criteria > between alpha and beta should be more open than the blocker criteria between > beta and rc. The idea is that we start accepting fewer bugs as blockers as > we get closer to RC. Every problem encountered can be evaluated along the > lines of: > > 1) Who is impacted? > 2) Is there a workaround? > 3) Is the workaround documented? > 4) Is the problem in the standard install path? > > And so on. I'm not saying we should compromise on release quality or > anything like that, but just start to ask more and more questions when > proposed blockers show up late. Is it really a blocker or not. > Again, that's not what I'm saying we should do. But if there's a traceback > that appears when someone is doing a LUKS installation to USB attached > storage that also has a pre-existing NTFS volume for "music"....that doesn't > sound like a really suport important use case late in the cycle. Maybe we > should consider documenting that as a limitation with or without a > workaround for that release and slating it for the next release. I have a better understanding of where you're coming from now, and particularly that you're basing this on a RHEL model (I didn't know this was how RHEL did blocker tracking). But I still disagree. :) I don't think my disagreement is necessarily a problem for you, though...let me see if I can put it a different way: Take the example you just gave. My position isn't 'we should take that bug as a blocker both four weeks out and one week out'. My position is 'we shouldn't take it as a blocker four weeks out _or_ one week out'. I'd phrase the problem more as 'there is a tendency in blocker review to accept things too casually early on'. We should only accept things as blockers that we really believe we should hold the release up for - even if it's six weeks until release and we know the bug is going to be fixed in two days, we should not accept it as a blocker unless it passes the 'if today was go/no-go and this was the last bug, would we block the release' smell test. Rather than throw almost everything on the blocker list when it's early and get more rigorous later on, I'd rather we just be rigorous the whole time. The blocker list isn't a work list (using the blocker list as a 'todo list' / reminder list is a tendency I've noticed in both RHEL and Fedora, and something I think we should fight), it is a list of bugs that must be fixed for the release to go out. No more, no less. If we wouldn't really block the release for a bug, it doesn't belong on the list at any point in time. I think we've been making a good effort to improve this in recent releases; it's not just an aspiration. So if there are concrete cases where you think bugs have been accepted as blockers that you wouldn't be comfortable taking as blockers the day before release, that's _always a problem_, and please highlight those bugs to us. I don't think it's 'not a problem' if we accepted them as blockers a long time before release and they got fixed before any chance of causing a slip, it's still a problem. We should stick firmly to our tight, minimal definition of what constitutes a blocker, regardless of the point in the release schedule. > We do want to make a good release and keep the users happy, but we need to > also remember that developers are people too and not just meat grinders for > bug reports. Sure, hence the need to keep the criteria realistic. > 1) Adjustments to the blocker acceptance criteria that I explained. > 2) Acknowledgement that concurrent releases are always in play for the > installer team. Generally two RHEL releases and one Fedora release. > RHEL gets priority. I would prefer to phrase 2) the way I already did: Fedora should ensure its release requirements are realistic in the context of the resources available. If the 'resources available' are to an extent defined by other commitments on the part of the anaconda team, then so be it, but I'd rather consider it in that context, rather than haul RHEL specifically into a Fedora context. > What I'm asking for is blocker acceptance criteria changes that are more > realistic given remaining time and other commitments. Or dump the > time/schedule requirement and just say we'll call it done when it's done. The schedule / blocker process balance is a bit of a delicate one, yeah. In practice we start getting a lot of pressure if the blocker process results in delays of several weeks. I think I'd like to look at this way, though: any time that following the release validation process properly results in a long slip, _that means something is wrong_. It usually means either that the release criteria are excessive, or it means that something went wrong in the feature process. I believe the feature process as it stands is not adequate in ensuring proposed features are really viable on the Fedora release schedule, and FESCo has also not always been assertive enough in dropping or delaying features that are clearly not meeting the schedule; I think sometimes problems are due to that, rather than to over-ambitious release requirements. > > To give a competing example, in anaconda 18.13 there is a bug in the new > > partitioning process - I call it 'guided partitioning', the dialog which > > attempts to help you free up space on a full disk, by deleting or > > shrinking partitions - which causes it to crash when trying to delete > > partitions. But if you go into the 'custom partitioning' interface you > > can successfully delete partitions. So by your principle, we would not > > take that bug as a release blocker. I don't think that would be a good > > decision: we should not release an installer which crashes when you try > > to follow the path you're guided to, for freeing up space to install the > > operating system. 'Don't do what the installer recommends you to do, > > instead go into this advanced process that's supposed to be for experts' > > is a workaround, and hence satisfies your requirement for not-a-blocker, > > but I really don't think it's a good story to tell people in the case of > > such a critical bit of functionality. > > Crashes through the standard install path would of course be a blocker. Why > would you think I would want otherwise? You're reading my proposal way too > literally. I'm afraid I have a tendency to read things literally, because processes and policies are written down, and if the person who wrote them is eaten by a raptor the only way you can reliably read them is 'literally' :) It makes drafting things a pain in the ass, but I think it makes what you wind up with a stronger process/policy. Your proposal didn't have escape clauses or exceptions, it simply read "Installer blockers should only be granted when there is no other way to accomplish the same task during installation." That's a very clear and unequivocal statement. We could not adopt it into the blocker evaluation process as it's written. As I said in my initial mail, I think in practice we mostly already do what you really want here, but we do need to document it more clearly. As the documentation stands, it really doesn't cover the question of workarounds much at all. (in reference to my points about development planning) > This is nothing new to us, we've been through it many times before. > But it doesn't mean we enjoy it. Unless we want to put anaconda in a > permanent maintenance mode, it means we will always be playing this > game. I don't think this is entirely correct. I don't want to get into specifics because I'm not plugged into the development planning process enough to make accurate criticisms, and I don't want to teach anyone's grandma to suck eggs, but to talk generally: Of course any project which involves major development will be more susceptible to bugs than any project which is solely in maintenance mode. But the _magnitude_ of the effect is certainly something that can be addressed by good planning. I don't want to harp on this too much as I'm sure it's something you're aware of, but we can certainly aim to be realistic and rigorous in our planning and timing of major new features. For instance, trying to ensure major developments are feature complete at Alpha time, so we're not still writing stuff during the Beta cycle. FESCo requiring much more detailed planning for feature submissions and being more decisive about rejecting/delaying features that are clearly lagging at Alpha stage. Stuff like that. > I didn't read this paragraph. This is a lot of text, but I've skimmed it. > What I think we should consider is whether Fedora should have a separate > entity or group that accepts blockers. Right now, that responsibility is > largely falling on your team. This isn't really the intent. The blocker meeting SOP - https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting - reads: "Blocker Bug Meetings are not owned by any one team in Fedora. They are a collaborative effort between Release Engineering, Quality Assurance, Development, and Project Management." and the intent is that, ideally, some or all of those groups should be represented at blocker review meetings. Lately, it seems that this hasn't been happening as much as it used to. We used to have fairly solid attendance from releng, FPL / FPM, and interested development parties. Recently it seems like these groups just aren't terribly interested in showing up and the meetings tend to be all or mostly QA people. And we go ahead and do the evaluations, because hell, someone has to. But as the procedure clearly states, it's not a QA-owned process, it's meant to be collaborative, and the onus is really on other interested parties to _show up and take part_. We do usually ping #anaconda when blocker review starts up and ask if anyone wants to take part; often no-one does. I think it's seen as taking away precious development time. It would be really nice if we starting getting more developers and Robyn and Jaroslav showing up for more blocker reviews. Admittedly, the meetings do get very long sometimes, but we've found it pretty hard to resolve that (reviewing 30 blockers is going to take a while, however you slice it). I do think it helps to have multiple perspectives, and that QA folks often tend to be more 'liberal' about accepting blockers than other groups; it would definitely be useful to have other groups present to act as a check on this. It might be an idea to start CCing the blocker meeting announcements out a bit more widely, to help with this. Or heck, we could also have some non-QA person run the meetings for a while, give it to FPL or FPM or someone. That would liven things up. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora http://www.happyassassin.net -- test mailing list test@xxxxxxxxxxxxxxxxxxxxxxx To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
