Keith, let me quickly respond to the first part of your email response. >> for cross area reviews all you need are IETF drafts. I doubt that reviewers are interested in digging through the Github issues, PRs, and mailing list discussions. > Well, the more working groups spread their discussions over various media, the harder it is for anyone not really tied up with the WG to contribute usefully. Some people may consider that a feature; I don't. > And in general, the later those who are not "in the loop" submit their input, the less effective that input is. Last Calls are the worst case, but reviewing the drafts are only slightly better. Really you need to get the broad input very early in the WG discussion. But you also > need people with broader interests watching the WG throughout its life to make sure their input is taken into consideration. I like your "romantic" view about high-quality reviews: There is this genius sitting someone. Nothing else to do he or she browses through the web and discovers the IETF website with all this interesting content. He or she reads through long email discussions and even longer drafts. Then, some earthshaking revelations are made and feedback is sent to the relevant working groups or draft authors. Joke aside - my experience with IETF feedback is very different. It is hard work and you cannot wait for a random outsider to provide high-quality feedback. You (as document authors, working group members, working group chair) have to reach out to other communities (like researchers, implementers, the deployment community, etc.). You have to give presentations at conferences and workshops. You have to organize interoperability events. Etc. You have to convince others that the work you are doing is worthwhile their time. In any case, our two views are just tiny data points on how reviews are done in the IETF and it would be interesting to investigate how we can improve the review process of our specifications. By that I am not talking about having another directorate being forced to review more documents. As you know, I have suggested to use formal methods more extensively during protocol design. I also think more implementation and interop testing would help to improve the quality of the specifications (and to involve others). Ciao Hannes PS: FWIW I agree with you that late reviews are less than ideal for the document authors and for the respective working group. However, doing an early review is pain for reviewers when documents are still inconsistent. Unless reviewers get some reward I doubt they are interested in suffering just for fun. IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
