Re: Review of Fedora 18 Release Criteria

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On Wed, 2012-10-10 at 12:17 -0400, David Cantrell wrote:

> You're not understanding what I was pointing out.  The blocker criteria
> between alpha and beta should be more open than the blocker criteria between
> beta and rc.  The idea is that we start accepting fewer bugs as blockers as
> we get closer to RC.  Every problem encountered can be evaluated along the
> lines of:
> 
> 1) Who is impacted?
> 2) Is there a workaround?
> 3) Is the workaround documented?
> 4) Is the problem in the standard install path?
> 
> And so on.  I'm not saying we should compromise on release quality or
> anything like that, but just start to ask more and more questions when
> proposed blockers show up late.  Is it really a blocker or not.

> Again, that's not what I'm saying we should do.  But if there's a traceback
> that appears when someone is doing a LUKS installation to USB attached
> storage that also has a pre-existing NTFS volume for "music"....that doesn't
> sound like a really suport important use case late in the cycle.  Maybe we
> should consider documenting that as a limitation with or without a
> workaround for that release and slating it for the next release.

I have a better understanding of where you're coming from now, and
particularly that you're basing this on a RHEL model (I didn't know this
was how RHEL did blocker tracking).

But I still disagree. :)

I don't think my disagreement is necessarily a problem for you,
though...let me see if I can put it a different way:

Take the example you just gave. My position isn't 'we should take that
bug as a blocker both four weeks out and one week out'. My position is
'we shouldn't take it as a blocker four weeks out _or_ one week out'.

I'd phrase the problem more as 'there is a tendency in blocker review to
accept things too casually early on'. We should only accept things as
blockers that we really believe we should hold the release up for - even
if it's six weeks until release and we know the bug is going to be fixed
in two days, we should not accept it as a blocker unless it passes the
'if today was go/no-go and this was the last bug, would we block the
release' smell test.

Rather than throw almost everything on the blocker list when it's early
and get more rigorous later on, I'd rather we just be rigorous the whole
time. The blocker list isn't a work list (using the blocker list as a
'todo list' / reminder list is a tendency I've noticed in both RHEL and
Fedora, and something I think we should fight), it is a list of bugs
that must be fixed for the release to go out. No more, no less. If we
wouldn't really block the release for a bug, it doesn't belong on the
list at any point in time. I think we've been making a good effort to
improve this in recent releases; it's not just an aspiration.

So if there are concrete cases where you think bugs have been accepted
as blockers that you wouldn't be comfortable taking as blockers the day
before release, that's _always a problem_, and please highlight those
bugs to us. I don't think it's 'not a problem' if we accepted them as
blockers a long time before release and they got fixed before any chance
of causing a slip, it's still a problem. We should stick firmly to our
tight, minimal definition of what constitutes a blocker, regardless of
the point in the release schedule.

> We do want to make a good release and keep the users happy, but we need to
> also remember that developers are people too and not just meat grinders for
> bug reports.

Sure, hence the need to keep the criteria realistic.

> 1) Adjustments to the blocker acceptance criteria that I explained.
> 2) Acknowledgement that concurrent releases are always in play for the
>    installer team.  Generally two RHEL releases and one Fedora release.
>    RHEL gets priority.

I would prefer to phrase 2) the way I already did: Fedora should ensure
its release requirements are realistic in the context of the resources
available. If the 'resources available' are to an extent defined by
other commitments on the part of the anaconda team, then so be it, but
I'd rather consider it in that context, rather than haul RHEL
specifically into a Fedora context.

> What I'm asking for is blocker acceptance criteria changes that are more
> realistic given remaining time and other commitments.  Or dump the
> time/schedule requirement and just say we'll call it done when it's done.  

The schedule / blocker process balance is a bit of a delicate one, yeah.
In practice we start getting a lot of pressure if the blocker process
results in delays of several weeks. I think I'd like to look at this
way, though: any time that following the release validation process
properly results in a long slip, _that means something is wrong_. It
usually means either that the release criteria are excessive, or it
means that something went wrong in the feature process. I believe the
feature process as it stands is not adequate in ensuring proposed
features are really viable on the Fedora release schedule, and FESCo has
also not always been assertive enough in dropping or delaying features
that are clearly not meeting the schedule; I think sometimes problems
are due to that, rather than to over-ambitious release requirements.

> > To give a competing example, in anaconda 18.13 there is a bug in the new
> > partitioning process - I call it 'guided partitioning', the dialog which
> > attempts to help you free up space on a full disk, by deleting or
> > shrinking partitions - which causes it to crash when trying to delete
> > partitions. But if you go into the 'custom partitioning' interface you
> > can successfully delete partitions. So by your principle, we would not
> > take that bug as a release blocker. I don't think that would be a good
> > decision: we should not release an installer which crashes when you try
> > to follow the path you're guided to, for freeing up space to install the
> > operating system. 'Don't do what the installer recommends you to do,
> > instead go into this advanced process that's supposed to be for experts'
> > is a workaround, and hence satisfies your requirement for not-a-blocker,
> > but I really don't think it's a good story to tell people in the case of
> > such a critical bit of functionality.
> 
> Crashes through the standard install path would of course be a blocker.  Why
> would you think I would want otherwise?  You're reading my proposal way too
> literally.

I'm afraid I have a tendency to read things literally, because processes
and policies are written down, and if the person who wrote them is eaten
by a raptor the only way you can reliably read them is 'literally' :) It
makes drafting things a pain in the ass, but I think it makes what you
wind up with a stronger process/policy. Your proposal didn't have escape
clauses or exceptions, it simply read "Installer blockers should only be
granted when there is no other way to accomplish the same task during
installation." That's a very clear and unequivocal statement. We could
not adopt it into the blocker evaluation process as it's written.

As I said in my initial mail, I think in practice we mostly already do
what you really want here, but we do need to document it more clearly.
As the documentation stands, it really doesn't cover the question of
workarounds much at all.

(in reference to my points about development planning)

> This is nothing new to us, we've been through it many times before.
> But it doesn't mean we enjoy it.  Unless we want to put anaconda in a
> permanent maintenance mode, it means we will always be playing this
> game.

I don't think this is entirely correct. I don't want to get into
specifics because I'm not plugged into the development planning process
enough to make accurate criticisms, and I don't want to teach anyone's
grandma to suck eggs, but to talk generally:

Of course any project which involves major development will be more
susceptible to bugs than any project which is solely in maintenance
mode. But the _magnitude_ of the effect is certainly something that can
be addressed by good planning. I don't want to harp on this too much as
I'm sure it's something you're aware of, but we can certainly aim to be
realistic and rigorous in our planning and timing of major new features.
For instance, trying to ensure major developments are feature complete
at Alpha time, so we're not still writing stuff during the Beta cycle.
FESCo requiring much more detailed planning for feature submissions and
being more decisive about rejecting/delaying features that are clearly
lagging at Alpha stage. Stuff like that.

> I didn't read this paragraph.  This is a lot of text, but I've skimmed it.
> What I think we should consider is whether Fedora should have a separate
> entity or group that accepts blockers.  Right now, that responsibility is
> largely falling on your team.

This isn't really the intent. The blocker meeting SOP -
https://fedoraproject.org/wiki/QA:SOP_Blocker_Bug_Meeting - reads:

"Blocker Bug Meetings are not owned by any one team in Fedora. They are
a collaborative effort between Release Engineering, Quality Assurance,
Development, and Project Management."

and the intent is that, ideally, some or all of those groups should be
represented at blocker review meetings. Lately, it seems that this
hasn't been happening as much as it used to. We used to have fairly
solid attendance from releng, FPL / FPM, and interested development
parties. Recently it seems like these groups just aren't terribly
interested in showing up and the meetings tend to be all or mostly QA
people. And we go ahead and do the evaluations, because hell, someone
has to. But as the procedure clearly states, it's not a QA-owned
process, it's meant to be collaborative, and the onus is really on other
interested parties to _show up and take part_. We do usually ping
#anaconda when blocker review starts up and ask if anyone wants to take
part; often no-one does. I think it's seen as taking away precious
development time. It would be really nice if we starting getting more
developers and Robyn and Jaroslav showing up for more blocker reviews.
Admittedly, the meetings do get very long sometimes, but we've found it
pretty hard to resolve that (reviewing 30 blockers is going to take a
while, however you slice it). I do think it helps to have multiple
perspectives, and that QA folks often tend to be more 'liberal' about
accepting blockers than other groups; it would definitely be useful to
have other groups present to act as a check on this.

It might be an idea to start CCing the blocker meeting announcements out
a bit more widely, to help with this. Or heck, we could also have some
non-QA person run the meetings for a while, give it to FPL or FPM or
someone. That would liven things up.
-- 
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net

_______________________________________________
Anaconda-devel-list mailing list
Anaconda-devel-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/anaconda-devel-list


[Index of Archives]     [Kickstart]     [Fedora Users]     [Fedora Legacy List]     [Fedora Maintainers]     [Fedora Desktop]     [Fedora SELinux]     [Big List of Linux Books]     [Yosemite News]     [Yosemite Photos]     [KDE Users]     [Fedora Tools]
  Powered by Linux