At 2:42 PM -0700 5/30/08, Suresh Krishnan wrote: > >I do share your sentiment, but this is not what we set out to do >originally. We set out to document the review process(es) and provide >some guidelines to effectively use the received reviews. I think the >right document to weave the stuff together would be the Tao. But I do >agree that adding references to the Tao and 2026 make sense. The >dangling reference to 2119 is an editorial oversight :-). If the Tao is the right document to do the weaving in, then the right thing to do may well be incorporate more discussion of the review process in the next version of the Tao. A stand-alone document which says "to get this context go read this first" might work, but it runs the usual risk. And statements like: "reviews also play an important part in how IETF determines consensus. " really need to be related to our process documents. Personally, I don't think that's actually either true or the point. Many solicited reviews are meant to discover technical show stoppers or areas where the impact of technology on other parts of 'net have not been taken into account. Determination of consensus is different, and it has a large amount to do with our participation model. Changes to the pool of folks from whom consensus is solicited should not be done lightly in informational documents. > > One of the things that I believe that anchoring should provide >> is a pretty significant change of perspective. As this reads now, >> it implies a lot of power in the hands of reviews to elicit (or even require) >> change. It seems to want document authors to accede to requests for >> tutorial material as a matter of course and to significant technical changes > >I am sorry you feel that way. It was not the intent of the document. The >document just states that a concern raised in the review deserves a >response (not necessarily a change). One valid response is "Mr. >reviewer, what you raised is not a problem. This was a design decision >taken by the WG and the discussion thread about this is at >http://mlarchive.invalid/msgfoo.html";. The tutorial material comment comes from this section: If the reviewer has not understood some part of the draft, a very common response is to explain the topic in email. However, often that explanation should really be in the draft itself, not just in emails to the authors, so that future readers will also understand the text. The key missing piece here is that the effort to get outside reviewers means that they are missing context that those actually using the document will likely have. Asking that it be included as a matter of course on the query of a reviewer produces changes that then have to be reviewed and agreed to by the working group. That's a lot of work for a lot of people for things which may well be obvious to those within the group using a technology. " The document's description of politeness in responses also sets timeframes that sound a lot like deadlines: "two or three business days might be a reasonable amount of time to take for a response. It is possible that the authors may be unavailable to respond to a review within this timeline since they may be sick/on vacation/busy with dayjob etc. In this case, the document shepherd (if any) can respond to the review and follow up with the authors at a later time." The ability to force an interrupt on busy people (either authors or shepherds) is real power, and we need to be careful how we cast that. A comment on a document (and that is what reviews are) does deserve to be considered, but the extent to which a response is required and the timeframe for that response is much more fluid than this implies. Bundling all comments received during a Last Call, considering them as an author team, then issuing a new draft or set of RFC Editor notes (or, honestly, punting them all) may be the most effective thing to do. In that case, the only early response you'll get is "message received". If you want that, permit me to suggest the handy MDN functionality likely available in your mail client. > > with a modicum of fuss. That's not the right approach. The outcome of our >> document development should not be a negotiation between the authors >> and the assigned reviewers. It should be a conversation in the working group >> among those who will actually develop the implementations, those who >> will deploy it, and those who are affected by the system of which the >> documented technology is a part. > >Agree. And hence this text in section 3.5 > >" After the author and reviewer agree that the issues have been fixed, > for working group documents, substantial issues need to be confirmed > on the mailing list. Once the document has entered IESG processing, > new versions should not be posted unless the document shepherd or AD > explicitly says so." You say "agree", but the text you point to says something substantially different from what I did. The text you point to says, in essence, "after the reviewer and document author have finished their negotiation, the mailing list should confirm". That's exactly wrong. > > Reviews that work to relate a particular document's technology >> to the larger whole of which it is a part (asking: how does this impact >> congestion control in the access network or core, use deployed security systems, >> relate to the identifier mechanisms common to URIs, etc.) are very valuable. >> But many reviews represent questions about decisions that come down to >> design choices that working groups should have the power to make without >> extensive second-guessing. Folks who want to have a say at the level can and >> should do so with the simple method of joining the working group list and commenting >> as part of the general development. That's not "review" (of someone else's work) >> it is "participation" (in joint work), and it is fundamentally more valuable. > >Agree in principle. But a "late" reviewer usually gets documents from >multiple wgs and subscribing to the MLs just to raise one issue can get >to be a tiring endeavor. Last year, I reviewed documents from 20 >different WGs (apart from the ones I am usually involved in). I cannot >handle mailing list traffic from 20 more mailing lists. I must not have been clear. I am not saying that a reviewer must subscribe to a mailing list to raise an issue. I am saying there are two classes of comments: 1) issues raised about the impact of the technology on the larger 'net or the usage of the technology within a larger context; 2) discussion of design decisions. Late comments of type 2 by folks who did not participate in the design discussions should expect fairly short replies; the right time and place to involve yourself in that is during the participation phase. If you deeply care that folks using ZNK use its ChillOut encoding when describing gttp extensions, you can join the gttp mailing list and argue the point. If you come as a reviewer and find they have chosen ZNK using its older BindMeTight encoding, you may ask something like "Were the trade-offs between BindMeTight and ChillOut considered?" and be satisfied with "Yep" (or prepared to help if they say "What's ChillOut?"). But if you push back on the use of BindMeTight, you had better be able to say how using it causes the protocol to fail to meet its goals. Anything less is not an appropriate late-stage review comment. It comes across as an attempt to subvert the process for attaining consensus. >If this were >mandated, I would rather not review the document. I would also like to >point out that most of the reviews received during IETF Last Call are of >the nature you describe. I tried to describe two different types, so it is not clear to me which you mean here. > > Without the context of how "participation" works, your documents description >> of "review" comes off very badly indeed. I hope that future versions can correct >> that and place review within a broader, more productive context. > >I personally feel that the TAO (4677 and descendants) should be the >document that sets the various pieces in context, but I am open to >suggestions on how to go about fitting this document into a broader >context. I have obviously not been clear here either, so permit this bluntness: as it stands now,this document is harmful. It fails to place the process in sufficient context. The result is a description that over-emphasizes late stage review and satisfying those who conduct those reviews. It needs to be re-written using a full context, folded into a document that establishes that context, or abandoned. Having it go forward alone, without that context, is not appropriate. To end on a high note, it is beautifully formatted. regards, Ted Hardie >Thanks >Suresh _______________________________________________ IETF mailing list IETF@xxxxxxxx https://www.ietf.org/mailman/listinfo/ietf