--On Tuesday, August 6, 2019 13:59 +0100 Stewart Bryant <stewart.bryant@xxxxxxxxx> wrote: > On 06/08/2019 13:48, Fernando Gont wrote: >> On 6/8/19 15:20, Stewart Bryant wrote: >>> >>> Looking for status = reported by Area is a good way of >>> seeing whether the ADs are processing them: >... >>> I am please to see that RTG is still low (though could be >>> lower!) considering the work it took to get them down to an >>> acceptable number a few years back, but some of the other >>> numbers look excessive. >> >> Skimming through some of the errata, some areas seem to have >> technical errata in their "Reported" Queue for over 5 years. >> >> Is there an upper bound on how long errata can remain >> unprocessed, or how long the queue may grow? >> >> Any plan to, say, limit the amount of time errata can remain >> unprocessed to, say, 6 months - 1 year? >> >> How does this play when an AD leaves the role with a full >> queue of unprocessed errata? Probably the same thing that happens when an erratum is reported long after the AD who managed the document or WG leaves the role. >... > The usual process is for the IETF Chair tell the ADs to > address them and get the numbers down, and for the rest of us > to have a discussion with the the Nomcom if neither of those > happen. Agreed, but let's take a step back. First, large numbers can also indicate that someone has gone on an errata rampage, finding many trivial issues and filing errata on them, possibly on the theory that it is an important contribution the IETF (sometimes it is). But if I were trying to draw strong inferences from statistics, I'd try to look at controlling for such efforts, especially in Areas where the subject matter makes it easier for a non-expert to read the documents and flag errors, especially editorial ones. More important (applying primarily to standards track documents, but generalizations may be possible)... What I'm about to say interacts with both the "evolving documents" (or whatever they are called this week) and WG declarations of stability discussions, so maybe things will change. But right now and historically (including those statistics), errata, even "verified" ones are nothing more than placeholders for some change that could occur in the future. "verification" generally doesn't indicate the consensus of a WG if there is a relevant one and certainly does not represent IETF consensus. So, ignoring errata reports that are just wrong, what we have are (1) many reports of editorial issues, some of which make the documents less clear and others that are just annoying (the distinction may depend on the reader's familiarity with the subject matter and/or comfort with suboptimal English and so may not be objective). (2) some reports of substantive problems where the document is unclear about what it is requiring or recommending or just plain wrong .. in both cases, at least in the opinion of the reporter and any reviewers of the erratum. These are all properly classified as "save for update" because, again, there isn't sufficient IETF consensus behind an erratum to say "the IETF has concluded that the document is incorrect". The _only_ solution to (2) and the best solution to (1) if the problem is significant is an Internet-Draft and, eventually, a new RFC, even if those documents say little more than "Section N.M of RFC XXXX is wrong and should read ..." (ideally with an explanation of why). An erratum is a useful step along the path to that result, but it not an endpoint. So, two questions: Is putting energy into incremental improvements in the errata system and errata processing worthwhile or would the time spent doing that work be better spent on updating or otherwise improved documents? And, if we want to measure errata processing and effectiveness, should we be also measuring how many errata that identify substantive issues lead to new documents on a timely basis? As part of the latter, if I need something to complain to the nomcome about an AD, should it be how many errata get processed or about how many documents get published and then generate substantive errata and/or how many errata that address substantive issues lead to new consensus documents. best, john
