On 7/30/11 11:05 AM, Joel M. Halpern wrote:
It seems to me that this does two things, both small but
useful. 1) It makes a minor change in the advancement
procedures so that they are more reasonable. They may still
not be sufficiently reasonable to be used, but it improves
them, and thereby improves the odds.
Actually, there is no evidence that this is an improvement. I'd
agree that it is a minor change, but there has been no serious
analysis of whether it is an improvement or not. To the extent
to which approving this lowers the odds of considering and
making changes that might actually have significant effects
(e.g., really sorting out what maturity levels mean in a world
of increasingly complex standards), it is harmful. See long
discussion below.
2) It is coupled to an
intent to actually behave according to what the document
says. Such an intent is obviously not feasible without some
change.
Well, yes and no. My sense of the discussion over the last year
is that a significant fraction of the community believes that
the critical path change in this area is an adjustment to the
threshold for Proposed Standard. That issue is addressed in
draft-bradner-restore-proposed, which requires no change to RFC
2026 or other formal procedures at all. It is not addressed in
this document.
It is useful to have our behavior and our documented
description of how we work match because the mismatch causes
confusion, at least for new participants, and sometimes even
for experienced participants.
I agree with this. But this document doesn't make nearly enough
of a contribution in that area to justify approving it. It
addresses exactly one provision of our processes in which
documentation and practice don't align, a provision about which
there is no subtlety or confusion within the community at all
(even though new participants may be confused). If the issue is
important, then let's make a serious effort to update the places
where 2026, 2028, 2031, 2360, 3710, 4071, and probably several
other documents have fallen at least somewhat out of line with
reality. If the particular issue of the annual review is really
more important than any of those other issues, I assume that a
stand-along document that changed it would rapidly achieve
community consensus (albeit with some complaints about wasted
time). Certainly it should not be sufficient to justify
approving this document... the change is almost irrelevant to
it.
It might be the case that it will improve the advancement
percentage. It might not. I can not imagine that it will
make that even worse.
I can because the effects of changes like this are actually very
hard to predict. It increases the requirements for the second
level, however slightly. Since we have trouble getting things
to the second level now, that increased requirement might reduce
incentives to bring things off Proposed at a least as much as a
theoretical "just one more level and then you are finished"
change would increase those incentives. By changing Proposed
from "1/3 of the way" to "1/2 way or a bit more", it might
remove a small incentive to take the additional step.
In addition, the "publication without a new RFC" provision may
actually further increase the de facto requirements for Proposed
Standards by requiring that those documents be edited to a high
standard of clear English and specification precision. While,
IMO, we have taken too little advantage of it, the current
provisions of RFC 2026 permit publishing a Proposed Standard as
soon as the WG understands it, leaving language cleanup (and
refined translation from the writing styles of many people in
the community whether native speakers of English or not) for
Draft Standard versions.
More important, for those of us who believe that a maturity
system based on rigid named categories that apply to entire
documents has outlived its usefulness as our specifications have
become more complex, approval of this document is almost certain
to cause consideration of such approaches to be postponed by
some years as people complain that the changes made in this
draft must have time to take effect and be evaluated.
---
A more extended analysis of other aspects of the situation with
this document follows. Those who don't like extended analyses
might want to stop reading at some point.
A very long time ago, a then-professor of mine observed that one
of the most common sources of failures in engineering,
especially computer engineering, was to look at a problem that
seemed too large or too intractable, find some easily-changed
and tractable part of the problem, do something with it (almost
anything), and then claim that significant progress had been
made on the original/ real problem because one had started on
it. In many cases, the approach actually makes the real
problem harder: systems become more complex, applying remedies
later turns out to be more complicated, and so on. The narrow
"solution" may also use up energy and creates expectations that
make it harder than it would have been otherwise to take on the
real work when the narrow job fails to solve any significant
problems.
An even more insidious version of the same approach is to
identify a problem and claim it is serious enough that one needs
to do _something_. One then proceeds to do something that has
nothing at all to do with the problem, just to demonstrate
motion (or in the hope that setting a butterfly in motion in one
part of the world will cause major change elsewhere).
These approaches are very different from taking a large design
issue and modularizing it into pieces that can be worked out
separately but with clear and well-understood interfaces. Doing
that right requires that the overall design issue be understood
well enough to define all of the modules and interfaces, not
doing one piece, ignoring the poorly-understood rest, and hoping
that things will sort themselves out.
The IETF periodically falls into this trap. We often see
protocols that are designed to handle the easy issues adopted
without consideration of the harder ones and edge cases, only to
discover that those protocols represent a framework into which
solutions for the other issues won't fit. The largest symptoms
of this used to show up only with Security issues -- protocols
being designed without security provisions with the expectation
that security would somehow be patched in later... and despite
repeated experiences that such patches are rarely completely
satisfactory. Now we see it spreading: I've seen "we knew
about those issues but ignored them and now have to clean up the
mess somehow" comments in two separate WGs in recent weeks and
suspect that there are many more. Many of the comments on this
list try to justify approving this draft on the same basis.
These "there is a problem so let's do something and hope it will
help", "even though we can demonstrate no logic that would
predict that the change being proposed will result in the
desired effect", and "let's fix something to show that we can"
approaches are _always_ justified on the basis of "baby steps"
or "incremental efforts". Those justifications would be quite
reasonable if we actually had good evidence of the linkages. In
the absence of such demonstrated linkages, we are just thrashing
around and wasting energy --energy that, under other
circumstances, could actually be used to address the problems.
So, while I applaud the efforts of the authors of this draft to
remove controversial and largely extraneous material, it seems
to me that the core issue remains the one that many people have
commented on in much earlier versions: At present, we don't get
many documents from Proposed Standard to Draft Standard, despite
the requirements for Draft Standard being slightly lower than
those set forth for Internet Standard in this document. We have
a possibly-serious problem with the norms for Proposed Standard
being too high in practice --far beyond what RFC 2026 requires.
It is noteworthy that draft-housley-two-maturity-levels no
longer makes any claim to solve that problem. On the other
hand, there has been considerable discussion in the community,
previously supported by the strongest advocates for the
predecessors of the current draft, that the norms for getting to
Proposed Standard _are_ the keystone problem. I don't
personally believe that -- I think things are more complicated--
but I do see it as a critical element (see
draft-bradner-restore-proposed (expired today but can be
reposted if that would be useful)).
So this draft now ignores the threshold problem for Proposed
Standard and argues that "Changing the Internet Standards
Process from three maturity levels to two is intended to create
an environment where lessons from implementation and deployment
experience are used to improve specifications". But there is no
logic to support the belief that intention will be satisfied.
The purpose of the three-level system is to create an
environment where the same lessons can be learned, applied, and
documented. This document does nothing to change or improve
that environment unless one believes that, magically, removing
the third level will increase motivations for getting things to
draft. The problem, which the document still describes, is that
documents don't move off the first level, not that somehow the
existence of the third one is pressing things down.
In progressing by baby steps, actual babies fall down a lot.
Usually they learn while falling and keep trying until they get
it right. Equally important, for their purposes, the important
target is learning how to walk more reliably, not reaching a
specific goal or solving a particular problem. The IETF lacks
those advantages: First, we are focused on the goal of making a
change and have posited that change in terms of a choice between
this particular document and doing nothing --all other
possibilities have been dismissed without significant
consideration. To quote one of the co-authors of this draft
from a different thread:
Herein lies the real problem: As with many process and
structure discussions in the IETF, folk often see only a
simplistic, binary choice between whatever they prefer, versus
something akin to chaos.
The world is more nuanced than that, and the choices more rich.
I agree, but this document is being treated as that sort of
binary choice.
Second, the community is easily exhausted by process debates and
the debate about this particular document has now been going on
for 13+ months. Even if the suggestion made at the plenary
about opening 2026 were serious, does anyone actually believe
the community will be able to take up and evaluate more
significant process changes in the near future? Or that this
document, if approved, will not be used to justify deferring
discussion of other changes because we have to see how it works
out?
So this is not "baby steps". It is one step, a step that makes
a change that isn't demonstrably connected to the problem it
purports to solve and that leads into a dead end.
john