On 4/12/2013 12:13 AM, Brian E Carpenter wrote:
Seeing randomly selected drafts as a Gen-ART reviewer, I can
say that serious defects quite often survive WG review and
sometimes survive IETF Last Call review, so the final review
by the IESG does serve a purpose.
Brian,
Of course it "serves" a purpose. The lack of perfection in the
specifications that reach the IESG has always been the justification for
the current review model.
But what is tiring about this line of justification is its continuing
failure to balance the analysis by looking at the costs and problems
that come with it. Does it provide regular and sufficient benefit to
justify its considerable costs?
First, it is a phenomenally expensive step, with a very damaging
organizational cost: the time and effort burden put on ADs makes the
pool of available ADs tiny, including dramatically reducing the range of
companies that can afford to donate senior (strategic) staff to it.
Along with this is the general issue that has triggered Joe's query:
the tendency of ADs to inject spontaneous, late-stage personal
engineering preferences, which might have been reasonable to add to the
original working group discussion but realistically have no place in a
final review. It's not that the preferences are necessarily
inappropriate, it's that they typically are not essential to the success
of the spec.
But in terms of the specific logic you've used, the presumption of the
"it catches errors sometimes" language is that the final output that is
the result is somehow perfect. Of course, that's a silly assumption.
But as soon as one acknowledges this, we are left with a model that must
class this as merely one more review in an imperfect sequence, with the
likelihood that an every new review will find more problems. Or we are
left with the view that pragmatics dictate accepting imperfections
because each step of the quality assurance model -- of which this is a
part -- really does need a strict cost/benefit analysis. This needs to
be done in terms of trade-offs, rather than an isolated approach that
counts the detection of an occasional problem as somehow an inherent and
controlling good.
An essential component to such a trade-off based model is recognizing
that the ultimate quality assurance step always has been, and remains,
the market. No amount of IETF q/a guarantees success. We have lots of
failures and we won't ever get to zero.
If the IESG review step really is essential, then we should show a
consistent pattern of its finding /essential/ errors that would have
caused failure in the field.
But if we can find that -- which I believe we cannot -- we have deeper
problems, of course, because that sort of shit needs to be found mch,
much sooner.
At base, the IESG review seeks to compensate for seriously inadequate
quality control during the development process...
d/
--
Dave Crocker
Brandenburg InternetWorking
bbiw.net