On 12/23/2015 09:21 PM, Dave Crocker wrote:
Anecdotal. Mine. Over enough years to represent a pattern. (I'm not alone in this, but I'm reporting my own experience) In 25 years, not one single RFC I've worked on had a serious problem caught by an AD, though many were eventually discovered to have serious problems
When I was an AD, I caught several serious problems with drafts after WG or IETF Last Call.
The problem I saw wasn't the inability to catch things so much as the inability to get them fixed. By the time a draft got past WG Last Call, it was generally so exhausted that it could not agree on how to make any significant change to a document, even in response to a legitimate and serious concern. In many cases, if there was a simple fix, it was quickly and happily implemented. But in far too many cases where there was a serious problem, the most that an IESG member could hope for was to get some weasel words inserted in the text or prepend an IESG note - neither of which really addressed the problem.
It was entirely too much work to do all of that review. After being on IESG for four years, it was about ten years before I could stomach a detailed review of any lengthy technical document. While on IESG I tried getting others to do reviews for me, but I found that I still had to read the documents under review to make sense of the reviews I was getting. If I had to do that again today, I'd probably try to give the reviewers more guidance.
But mostly I think that WGs need to have their work subject to formal external review much earlier in the process than Last Call, particularly when the WG's work has the potential to impact other WGs or other areas.
Keith