On 6/8/19 16:55, John C Klensin wrote: > > > --On Tuesday, August 6, 2019 13:59 +0100 Stewart Bryant > <stewart.bryant@xxxxxxxxx> wrote: > >> On 06/08/2019 13:48, Fernando Gont wrote: >>> On 6/8/19 15:20, Stewart Bryant wrote: >>>> >>>> Looking for status = reported by Area is a good way of >>>> seeing whether the ADs are processing them: >> ... >>>> I am please to see that RTG is still low (though could be >>>> lower!) considering the work it took to get them down to an >>>> acceptable number a few years back, but some of the other >>>> numbers look excessive. >>> >>> Skimming through some of the errata, some areas seem to have >>> technical errata in their "Reported" Queue for over 5 years. >>> >>> Is there an upper bound on how long errata can remain >>> unprocessed, or how long the queue may grow? >>> >>> Any plan to, say, limit the amount of time errata can remain >>> unprocessed to, say, 6 months - 1 year? >>> >>> How does this play when an AD leaves the role with a full >>> queue of unprocessed errata? > > Probably the same thing that happens when an erratum is reported > long after the AD who managed the document or WG leaves the role. THere seems to be a difference there. What I was referring to is that if an AD leaves the role with 2 years worth of errata unprocessed, that's probably an interesting "fresh start" for the incoming AD... >> ... >> The usual process is for the IETF Chair tell the ADs to >> address them and get the numbers down, and for the rest of us >> to have a discussion with the the Nomcom if neither of those >> happen. > > Agreed, but let's take a step back. First, large numbers can > also indicate that someone has gone on an errata rampage, > finding many trivial issues and filing errata on them, possibly > on the theory that it is an important contribution the IETF > (sometimes it is). But if I were trying to draw strong > inferences from statistics, I'd try to look at controlling for > such efforts, especially in Areas where the subject matter makes > it easier for a non-expert to read the documents and flag > errors, especially editorial ones. For technical errata, I'd actually try to foster it. It's the equivalent to "code review": you do want people to search for bugs. That tends to improve things. > More important (applying primarily to standards track documents, > but generalizations may be possible)... > > What I'm about to say interacts with both the "evolving > documents" (or whatever they are called this week) and WG > declarations of stability discussions, so maybe things will > change. But right now and historically (including those > statistics), errata, even "verified" ones are nothing more than > placeholders for some change that could occur in the future. > "verification" generally doesn't indicate the consensus of a WG > if there is a relevant one and certainly does not represent IETF > consensus. > > So, ignoring errata reports that are just wrong, what we have > are > > (1) many reports of editorial issues, some of which make the > documents less clear and others that are just annoying (the > distinction may depend on the reader's familiarity with the > subject matter and/or comfort with suboptimal English and so may > not be objective). > > (2) some reports of substantive problems where the document is > unclear about what it is requiring or recommending or just plain > wrong .. in both cases, at least in the opinion of the reporter > and any reviewers of the erratum. These are all properly > classified as "save for update" because, again, there isn't > sufficient IETF consensus behind an erratum to say "the IETF has > concluded that the document is incorrect". If only we were to revise specs frequently. That doesn't seem to be the case. And in fact, there's people that find that undesirable... > The _only_ solution to (2) and the best solution to (1) if the > problem is significant is an Internet-Draft and, eventually, a > new RFC, even if those documents say little more than "Section > N.M of RFC XXXX is wrong and should read ..." (ideally with an > explanation of why). An erratum is a useful step along the path > to that result, but it not an endpoint. I agree that errata are not and endpoint. On the other hand, documents that patch other documents tend to end up in a mess. Because when you read the base spec, you have to remember what parts have been obsoleted, do a mental replacement, etc. When the number of updating documents (or the number of parts of the original document that need to be patched) becomes meaningful, the right thing would be to coalesce all into a single document. > So, two questions: Is putting energy into incremental > improvements in the errata system and errata processing > worthwhile or would the time spent doing that work be better > spent on updating or otherwise improved documents? One would expect that the two are not mutually exclusive. Many wgs benefit from the errata processing, such that when a document is revised, part of the bis effort is to incorporate all verified errata. > And, if we > want to measure errata processing and effectiveness, should we > be also measuring how many errata that identify substantive > issues lead to new documents on a timely basis? Yes, we should. Also low numbers for such stats could indicate either not-so-meaningful errata, but also wgs simply failing to revise documents. > As part of the > latter, if I need something to complain to the nomcome about an > AD, should it be how many errata get processed or about how many > documents get published and then generate substantive errata > and/or how many errata that address substantive issues lead to > new consensus documents. All? If RFCs were code, errata would be bug reports. Failure to process and patch bug reports is bad. Shipping buggy code is not nice, but also happens. Ideally, you'd expect that code is generally ok, but if a bug is found, it is patched in a timely manner. Same for RFCs: one would expect that the quality is high, and there's not much room for filing errata. BUt if errors are found, that such error reports are processed in a timelier manner. Thanks, -- Fernando Gont SI6 Networks e-mail: fgont@xxxxxxxxxxxxxxx PGP Fingerprint: 6666 31C6 D484 63B2 8FB1 E3C4 AE25 0D55 1D4E 7492
