On 5/30/08 7:54 PM, Ted Hardie allegedly wrote: >> There are several issues that get mixed up together in defining a late >> external review process. By definition, this is an external review. So >> it is not reasonable to say that the reviewer should think like a WG >> member, or have the full email conversations available. > > No, it is not. But it is also not reasonable to assume that each document > carry the full context of an area with it. There is a balance there, > and assuming that contextualizing information should be added to > each document on the basis of reviewer request is a problem. In addition > to making each document very large, they run the great risk of getting > out of synch, with the usual hilarity resulting. If there is _written_ context in other drafts, that's fine. If _cultural_ context is missing from the drafts, that's a problem that should be noted and fixed. Sometimes the drafts where the context should be added are already out the door, and the only place to put the needed context is in the draft under review. > But we're not writing our documents for reviewers; we're writing them for > implementors, operators, and our successors, toiling in these protocol farms, > right? So there is some level of common knowledge among those working > within a protocol area we can and should assume in our documents. You can't assume that. We get strange questions from implementors and operators all the time. Far stranger than the ones from reviewers. Better to assume that reviewers understand the audience as well as you do and what their needs will be. > We can and should provide citations (that's what those nifty normative > and informative references are for), but that doesn't translate into providing > tutorial text. I was once a big fan of it, but I see people go wrong on the basis > of thrown-in tutorial text often enough now that I think it is often > less than helpful and sometimes actively harmful (by causing the reader > to believe that it is sufficient context). Personally I agree in separating out tutorial text, but all context required must be supplied somewhere. On 6/2/08 3:00 PM, Ted Hardie allegedly wrote: > A review that is raising a showstopper has a provable or disprovable statement > in it. "This *will not work* in the following scenario" or even "This seems > to have poor results in networks with high rates of non-congestive loss" creates the > opportunity for the reviewer and working group to discuss the issue in terms > that can be tested. Agreed, and reviewers with expertise do this. Reviewers with less expertise may just explain their doubt and ask if it is justified, but even questions like that should be considered if they have not been already, because on occasion I've seen them save a group from disaster. > But review comments that do not contain testable assertions end up > being subjective. "I think ZNK ChillOut is better than ZNK BindMeTight, > for the following reasons" may contain a good set of reasons, but it > should never over-rule the consensus of a working group that has > agreed on ZNK BindMeTight unless one of those reasons amounts to > "it doesn't work". In many WGs there are multiple designs that would work, and various tradeoffs are made in choosing one. There is never (afaik) a requirement that one approach prove the other simply does not work. Similarly early reviews can be very helpful even in re-examining known tradeoffs or bringing up new ones so that the balance between the approaches changes. It is rare for a late review to change a WG decision in this way -- it's hard for a WG to backtrack -- but it does happen occasionally, and there is never a requirement that a review prove that a design is utterly broken, just that there are serious disadvantages. If a reviewer comment is opinion based on a great deal of experience, for example with system design, then the reviewer -- presenting him/herself as having special expertise -- then has an obligation to recommend a solution to the problem he/she sees, in detail. > Comments of the type "I think Section 4 is not > clear enough for an implementor to follow" are also subjective; they > are very valuable and may serve as a guide to the WG/author team, > but it is important for the reviewer to recognize that they may not > result in change. Yes. Scott _______________________________________________ IETF mailing list IETF@xxxxxxxx https://www.ietf.org/mailman/listinfo/ietf