The publication process has human beings (ADs but also authors & shepherds) in the loop in order to provide a judgement call on the changes at any stage (up to AUTH48). And, like Ben wrote, the current IESG members tend to err on the cautious side, i.e., request another Last Call by the community (and explicit forward to the originating WG if applicable).
On the diff side, personally, I prefer to rely on a diff tool than on the authors' change log.
So, John's concern is valid, but I think to process already cares about it.
Regards
-éric
On 25/06/2022, 23:06, "iesg on behalf of Brian E Carpenter" <iesg-bounces@xxxxxxxx on behalf of brian.e.carpenter@xxxxxxxxx> wrote:
(Front posting to avoid a TL;DR.)
It's already the case that if the AD considers that the changes after Last Call and IESG review are substantive, a second Last Call can (and should) be issued. Isn't that sufficient? It does rely on the AD's judgment, of course, and should probably be done more often.
I agree about the change log, although I tend to rely on rfcdiff or iddiff.
Regards
Brian Carpenter
On 26-Jun-22 04:49, John C Klensin wrote:
> IESG,
>
> While neither issue is new, I've been troubled in the last few
> weeks by a pair of related issues. I want to identify them (as
> briefly as I can) and, while it won't solve them, make some
> small practical suggestions. I am writing about WG documents
> but, especially with various interpretations of RFC 8789,
> AD-sponsored individual ones may be even more subject to
> problems.
>
> In the last few steps of our document approval process, an IETF
> Last Call is announced and announced about a specific version of
> a document. If there are useful (typically as judged by the
> authors, not, e.g., the WG that is responsible for the document)
> comments during Last Call, the document is revised during and/or
> immediately after that period, sometimes more than once. After
> the Last Call period ends, the IESG starts its formal review, in
> theory making a determination of community consensus based on
> the Last Call comments. In practice, the ADs often find
> issues not identified during Last Call and their comments often
> lead to further document changes before a draft (possibly with
> notes) goes off to the RFC Editor. Over the years, some of the
> issues caught in IESG review have been very important so the
> present model, as practiced, is is probably nearly as good as we
> can do. However, it raises two very fundamental questions about
> IETF consensus (rough or otherwise) about the finished work:
>
> (i) Is the document on which the IESG review starts
> substantively the same as the one on which the Last Call was
> started? Would any of the changes have elicited addition
> substantive community comments had the been in the document when
> the Last Call started? Presumably both questions are answered
> by the responsible AD in allowing the document to move into IESG
> evaluation, but that means one person, with an investment in the
> document and the WG, is making a potentially significant
> decision on behalf of the community.
>
> (ii) When changes are made to the document at IESG request (or
> even late discovery of issues by the authors), does the document
> forwarded to the RFC Editor still represent community consensus?
> Is that question even explicitly asked in all cases?
>
> These two stages of the process are, at least in principle, even
> more important than the AUTH48 one that has drawn recent
> attention. I don't think there is a "solution" to either, but
> have some suggestions to mitigate the possible risks of a
> document being published that does not represent community
> consensus. AFAICT, none of them require revisions to normative
> process documents or rules.
>
> (1) While it has never been a requirement (and, IMO, should not
> be), many WGs and authors have included "Change Logs" that
> summarize differences among versions of an I-D. If new versions
> of an I-D are produced after the Last Call starts, the IESG
> should insist on such summaries if the modified versions are to
> be considered without restarting the Last Call using those
> versions. That would allow community members who have already
> reviewed and commented on a previous version, are working on
> comments, and even those who have decided they do not need to
> post Last Call comments based on a earlier version to swiftly
> determine whether they need to reexamine the I-D or particular
> sections of it. While an irresponsible or malicious editor
> could game that requirement in the hope of slipping something
> through, that risk is better dealt with by choices of editors
> (or firing them if needed) than by eliminating the benefits of
> responsible behavior.
>
> (2) While there are many advantages of making changes to
> documents by Github pull requests, they do not make it easy for
> someone concerned about aspects of a document but without the
> time to get fully drawn into the work (IMO, exactly the people
> IETF Last call is intended to reach because those who are more
> involved presumably would have been part of the WG) to
> understand all changes in context. Probably there should be a
> clear requirement that significant changes (e.g., replacement or
> additions of whole sections or appendices) be accompanied by a
> new draft so that it is possible to do a diff. Even more
> important, if _any_ changes were made during Last Call (or
> during AD review before IESG review starts), a new version
> should be posted so that the community can be aware of exactly
> what the IESG is evaluating. While I am less concerned about
> it, it would be good for the IESG to consider whether a new
> draft should be required along with a Protocol Action or
> Document Action notice rather than reflecting any but the most
> trivial of correction in IESG notes to the RPC.
>
> (3) One way to mitigate some of the perceived problems would be
> for the IESG to be much more ready to push documents back to the
> originating WGs when significant issues are raised or changes
> made, requiring at least a brief WG Last Call. That would
> assure that there is at least still WG consensus around the
> post-Last Call version of a document the WG presumably produced.
> It would obviously add an extra week or two to the process when
> things go smoothly and so should be used with discretion. But
> it would, in some (many?) cases be faster and more consistent
> with fairness than an extended discussion or negotiation between
> authors and IESG members. It would also be consistent with a
> view that, if significant changes are needed to a document after
> Last Call starts, they are often indicative of insufficient
> diligence on the part of the WG to be sure that necessary
> substantive issues have been reviewed and considered before
> publication is requested.
>
> (4) AFAICT, current procedures allow any member of the community
> to appeal a decision by the IESG to start evaluating/voting on a
> document on the grounds that too many changes were made during
> (or after) Last Call to permit a fair judgment of community
> consensus. Similarly, a Protocol or Document action notice
> could be appealed on the grounds that too many substantive
> changes have occurred (either since the IESG evaluation or the
> Last Call started) to be consisting with fairness in assessing
> community consensus. I do not believe either of those appeal
> opportunities has been abused since RFC 2026 was published in
> 1996. It might improve both fairness and opportunities for
> quick airing and resolution of issues if the IESG created
> explicit (to the community, not just IESG members and careful
> watchers) benchmarks for "starting IESG evaluation" and "about
> to go to the RFC Editor" with short windows of opportunity for
> "maybe not quite yet" comments at those stages (perhaps quick
> notes to the Last Call list). They would not eliminate the
> possibility of appeals and could presumably be overlapped with
> other activities (e.g., IESG agenda setting) but might provide a
> more reliable and less time-consuming way to identify any issues
> that had arisen after the WG's publication request.
>
> Those suggestions are independent of each other -- if some do
> not resonate, please consider the other(s). FWIW, I believe
> that, in addition to improving fairness and the quality of the
> assessed IETF consensus and the documents themselves, they would
> also make the IESG review process more efficient and reduce
> burdens on IESG members.
>
> thanks,
> john
>
