(Front posting to avoid a TL;DR.) It's already the case that if the AD considers that the changes after Last Call and IESG review are substantive, a second Last Call can (and should) be issued. Isn't that sufficient? It does rely on the AD's judgment, of course, and should probably be done more often. I agree about the change log, although I tend to rely on rfcdiff or iddiff. Regards Brian Carpenter On 26-Jun-22 04:49, John C Klensin wrote:
IESG, While neither issue is new, I've been troubled in the last few weeks by a pair of related issues. I want to identify them (as briefly as I can) and, while it won't solve them, make some small practical suggestions. I am writing about WG documents but, especially with various interpretations of RFC 8789, AD-sponsored individual ones may be even more subject to problems. In the last few steps of our document approval process, an IETF Last Call is announced and announced about a specific version of a document. If there are useful (typically as judged by the authors, not, e.g., the WG that is responsible for the document) comments during Last Call, the document is revised during and/or immediately after that period, sometimes more than once. After the Last Call period ends, the IESG starts its formal review, in theory making a determination of community consensus based on the Last Call comments. In practice, the ADs often find issues not identified during Last Call and their comments often lead to further document changes before a draft (possibly with notes) goes off to the RFC Editor. Over the years, some of the issues caught in IESG review have been very important so the present model, as practiced, is is probably nearly as good as we can do. However, it raises two very fundamental questions about IETF consensus (rough or otherwise) about the finished work: (i) Is the document on which the IESG review starts substantively the same as the one on which the Last Call was started? Would any of the changes have elicited addition substantive community comments had the been in the document when the Last Call started? Presumably both questions are answered by the responsible AD in allowing the document to move into IESG evaluation, but that means one person, with an investment in the document and the WG, is making a potentially significant decision on behalf of the community. (ii) When changes are made to the document at IESG request (or even late discovery of issues by the authors), does the document forwarded to the RFC Editor still represent community consensus? Is that question even explicitly asked in all cases? These two stages of the process are, at least in principle, even more important than the AUTH48 one that has drawn recent attention. I don't think there is a "solution" to either, but have some suggestions to mitigate the possible risks of a document being published that does not represent community consensus. AFAICT, none of them require revisions to normative process documents or rules. (1) While it has never been a requirement (and, IMO, should not be), many WGs and authors have included "Change Logs" that summarize differences among versions of an I-D. If new versions of an I-D are produced after the Last Call starts, the IESG should insist on such summaries if the modified versions are to be considered without restarting the Last Call using those versions. That would allow community members who have already reviewed and commented on a previous version, are working on comments, and even those who have decided they do not need to post Last Call comments based on a earlier version to swiftly determine whether they need to reexamine the I-D or particular sections of it. While an irresponsible or malicious editor could game that requirement in the hope of slipping something through, that risk is better dealt with by choices of editors (or firing them if needed) than by eliminating the benefits of responsible behavior. (2) While there are many advantages of making changes to documents by Github pull requests, they do not make it easy for someone concerned about aspects of a document but without the time to get fully drawn into the work (IMO, exactly the people IETF Last call is intended to reach because those who are more involved presumably would have been part of the WG) to understand all changes in context. Probably there should be a clear requirement that significant changes (e.g., replacement or additions of whole sections or appendices) be accompanied by a new draft so that it is possible to do a diff. Even more important, if _any_ changes were made during Last Call (or during AD review before IESG review starts), a new version should be posted so that the community can be aware of exactly what the IESG is evaluating. While I am less concerned about it, it would be good for the IESG to consider whether a new draft should be required along with a Protocol Action or Document Action notice rather than reflecting any but the most trivial of correction in IESG notes to the RPC. (3) One way to mitigate some of the perceived problems would be for the IESG to be much more ready to push documents back to the originating WGs when significant issues are raised or changes made, requiring at least a brief WG Last Call. That would assure that there is at least still WG consensus around the post-Last Call version of a document the WG presumably produced. It would obviously add an extra week or two to the process when things go smoothly and so should be used with discretion. But it would, in some (many?) cases be faster and more consistent with fairness than an extended discussion or negotiation between authors and IESG members. It would also be consistent with a view that, if significant changes are needed to a document after Last Call starts, they are often indicative of insufficient diligence on the part of the WG to be sure that necessary substantive issues have been reviewed and considered before publication is requested. (4) AFAICT, current procedures allow any member of the community to appeal a decision by the IESG to start evaluating/voting on a document on the grounds that too many changes were made during (or after) Last Call to permit a fair judgment of community consensus. Similarly, a Protocol or Document action notice could be appealed on the grounds that too many substantive changes have occurred (either since the IESG evaluation or the Last Call started) to be consisting with fairness in assessing community consensus. I do not believe either of those appeal opportunities has been abused since RFC 2026 was published in 1996. It might improve both fairness and opportunities for quick airing and resolution of issues if the IESG created explicit (to the community, not just IESG members and careful watchers) benchmarks for "starting IESG evaluation" and "about to go to the RFC Editor" with short windows of opportunity for "maybe not quite yet" comments at those stages (perhaps quick notes to the Last Call list). They would not eliminate the possibility of appeals and could presumably be overlapped with other activities (e.g., IESG agenda setting) but might provide a more reliable and less time-consuming way to identify any issues that had arisen after the WG's publication request. Those suggestions are independent of each other -- if some do not resonate, please consider the other(s). FWIW, I believe that, in addition to improving fairness and the quality of the assessed IETF consensus and the documents themselves, they would also make the IESG review process more efficient and reduce burdens on IESG members. thanks, john
