Trying to paraphrase Ted's response to my comments, and then reply. (The paraphrase is to try to make sure that I understood the comment.) I think that Ted is saying that review comments should have testable, actionable comments. Further, if I am reading him right, he is saying that comments that are not testable are probably opinion. That does seem a good place to start. Further, Ted comments taht reviewers should understand that not all their comments are going to be accepted. I agree. Not even all of their comments based on proper understanding of the document are going to be accepted. But, the question of where the boundary is between "that does not work" and "in my opinion that is a bad idea" is, in my experience, frequently fuzzy. To use a concrete example, we have plenty of experience to show that using a field length to determine what kind of field something has is a bad idea. It fails, over time, in a myriad ways. But no matter how many horror stories I put together, I was going to lose that particular debate with that particular document. On the other hand, there are things which look perfect in theory, but we know turn out to be more fragile and are therefore bad ideas. The assertion that such a thing is a bad idea is going to look like opinion, but is useful. In fact, part of my concern is that sometimes opinions can be as important as facts. Yes, reviewers need to be prepared to accept push-back. But it should not be against the rules for a reviewer to say "I have seen exactly this before, and by the time they made it work it was a mess. Please don't do this." And even if it is opinion, it deserves a response. TO use another example, when I said to a working group "Having worked with a lot of testers, I think that they will have trouble following the instructions as you have written them." I was clearly stating an opinion. But it was an opinion that at least deserved a response. (In fact, the working group did respond, and a good compromise was reached on that and a number of related issues.) Part of this is that just as we expect authors to respond politely to reviewers, reviewers have an obligation of politeness towards the WG. Reviewers should assume that odd things are there for good reasons, even while raising the question as to whether they belong. But trying to create a hard line between opinion and fact is I believe counter-productive. If the issue is larger than it looks on reading the review, then the reviewer can add more explanation, examples, or justification in the dialog with the authors / WG. Similarly, if the WG is overreacting to something but feels it is a big deal, the reviewer can say in followup "I had not realized that would be a big deal. Let it be." We are all presumably adults. We are also human. There is also a subtext that comes up periodically about the comments being actionable. This has been raised for outside reviews and IESG reviews. While I certainly agree that actionable comments are much better, it is not always possible. If the problem is that the reader can not figure out what the text means, it is hard to suggest a fix. It also gets laborious if the text could mean one of three different things. Should the reviewer have to provide text for all three alternative meanings? After all "clarify the text" while clear is not very specific as to what action is needed. There are also other cases that come up where it may be impractical to require actionable comments. Yours, Joel M. Halpern Ted Hardie wrote: > At 5:12 PM -0700 5/30/08, Joel M. Halpern wrote: >>> These both sound like excellent reviews: you expressed your personal >>> design preferences in the first instance but did not try to force it over >>> the consensus of the working group, and pointed out a showstopper >>> in the second. >>> >>> Now, show me in this draft how these two cases are distinguished, >>> which is critical to getting a review done right? >> The problem I have is that I do not know how to write text in a draft >> that distinguishes those two. The line between them is very tricky, and >> possibly subjective. > > Here is where I think we may have some room for some fruitful discussion. > > A review that is raising a showstopper has a provable or disprovable statement > in it. "This *will not work* in the following scenario" or even "This seems > to have poor results in networks with high rates of non-congestive loss" creates the > opportunity for the reviewer and working group to discuss the issue in terms > that can be tested. The working group may argue (and, presumably, demonstrate) > that either statement is false, but there is a way to move forward that is not > just iteration of position. To put that task to the working group, a reviewer > should be able to back their statement with a reasonably clear, worked set of > reasons for the assertion. A working group can disprove the statement, > indicate why an applicability statement is a better response ( demonstrating > that the cases where it does work or does have reasonable results are useful > enough and distinguishable enough that the document should go forward), > or make relevant technical changes. > > But review comments that do not contain testable assertions end up > being subjective. "I think ZNK ChillOut is better than ZNK BindMeTight, > for the following reasons" may contain a good set of reasons, but it > should never over-rule the consensus of a working group that has > agreed on ZNK BindMeTight unless one of those reasons amounts to > "it doesn't work". Comments of the type "I think Section 4 is not > clear enough for an implementor to follow" are also subjective; they > are very valuable and may serve as a guide to the WG/author team, > but it is important for the reviewer to recognize that they may not > result in change. > >> And part of the problem is to avoid turning it into a fight. If all >> review comments get clear, reasonably timely responses, there is room >> for the discussion without acrimony. > > I don't think that helps unless there is a clear set of expectations > *on the part of reviewers* on what their responsibilities are in > producing an actionable review and in accepting that some of their > comments may result in no change. I believe that the right way > to do that is to ground the description of the review process in > a strong understanding of the document production and participation > model of the IETF. If you don't, you end up with an unbalanced > view in which the few hours of effort put in on a solicited review > have a very much larger effect than the effort of the participants > who produced the document and will use the protocol it > describes. > >> Would it address your concern if the document said something like: >> "Reviewers should be sensitive to the difference between >> their personal opinions (and preferences) and issues >> which will affect the correct operation or interoperation >> of the documents under review" >> ? > > I don't think this is nearly enough. > > Ted > > >> I have no problem with pointing out that there are two different >> categories. I have real problems with trying to define a hard line >> which distinguishes them. >> >> Yours, >> Joel >> >> PS: While there are other differences in our views, they seem to be >> questions on which reasonable folk may differ and we can let the >> community sort out how to write the wording. >> _______________________________________________ >> IETF mailing list >> IETF@xxxxxxxx >> https://www.ietf.org/mailman/listinfo/ietf > > _______________________________________________ IETF mailing list IETF@xxxxxxxx https://www.ietf.org/mailman/listinfo/ietf