Dear Jari;
On Jun 25, 2008, at 7:37 AM, Jari Arkko wrote:
Bernard, Russ,
I changed the subject line, I think the thread has continued long
enough :-) Indeed, I collect a set of measurements. These are based
on pulling information from the tracker and the documents. The
reason for setting this up was to try to better understand what is
happening in the IETF process at various levels: for each document,
my own AD work, how the IESG works, where is the time spent in the
IETF process as a whole.
http://www.arkko.com/tools/admeasurements/ (start page and
disclaimers)
http://www.arkko.com/tools/admeasurements/stat/base.html (main IESG
measurements)
http://www.arkko.com/tools/lifecycle/draft-ietf-mobike-protocol-timing.html
(exists for each approved draft)
The data is current and is being updated every week or so. But
before everyone looks at the graphs and jumps to conclusions, I
wanted to insert a few disclaimers. First, this is based on tracker
data and it does not go into that far into the past -- and some of
the older data might even not be all that accurate. Second, I
recently added a lot of measurements and I don't yet have 100%
confidence that my tool gets everything correct. This is
particularly true of the items looking at various aspects of
Discusses. Third, some of the measurements look at tracker
substates, which are not used by all ADs and no one uses them
completely accurately. And last but not least, I don't want anyone
to blindly look at the numbers without considering what's behind
them. There are significant variations between documents. Tracker
status and real world situation may be different. Areas differ.
There is limited amount of information about documents prior to them
becoming WG documents. And its not necessarily the goal of the IETF
to produce a large number of documents as fast as it can; I have no
way to measure quality in these numbers. These are so hard issues
that its not even clear that the statistics have more value than
acting as entertaining trivia about the process.
That being said, it is beneficial to understand what is happening
and what changes are occurring in the process. Or understand where
improvements might have a negligible vs. visible impact.
One of the things that the IESG has been concerned about recently is
that the number of outstanding Discusses is growing. We talked about
this in our May retreat and identified some actions to help reduce
this problem. For instance, better tool support so that the ADs
would better see the different things that are waiting for their
action, getting the document shepherds better involved in the
resolution process, informing authors how they should respond to
Discusses, using RFC Editor notes to make small changes when the
authors are slow in producing a new version, better checks of
documents before they come to telechats (e.g., to ensure that formal
syntax in a document is free of errors), etc. These would be in
addition to the usual things we'd do, like debate whether the
Discuss was within the Discuss criteria, whether the issue is real,
try to ensure that the AD and the authors are being responsive over
e-mail, etc.
Another interesting area to think about is the time that our
processes take. For instance, documents that go through me take on
the average five months from publication request to approval. But
there is a lot of variation. This time includes everything from AD
review, IETF Last Call to IESG review and waiting for document
revisions, etc. One interesting piece of information is that
documents that require no revision during this process are very
rare, but they go through the system about five times as fast. If we
look at the (unreliable) substates, they appear to indicate that the
IESG processing time is divided roughly 1:2:1 for waiting on AD,
authors, and mandatory process waits like last call periods.
I am working to extend the analysis a little bit further by
including individual draft and WG document stages. You see some of
the results of this in the third URL above, but I'd like to
understand what fraction of the overall draft-smith-00 to RFC time
is taken by the different stages for all IETF work, and how the
stages have developed over time.
Comments and suggestions on what would be useful to measure are
welcome.
I, at least, always like looking at such statistics, so I thank you.
In my opinion, the real value of this come from trends. The IESG
clearly should not have
Soviet style work norms, for a bunch of reasons, and it may not be
possible to say what an approval rate should be,
but if something changes greatly over time it is generally worthwhile
to figure out why.
To that end, I frequently find it useful to put means and standard
deviations as horizontal lines on the plot, at least once enough data
is available (say, 5 data points). (You can do the mean as a solid
line and the mean + and - one or two standard deviations as dashes or
dots or a fainter line).
The assumption is that there are underlying random (gaussian or
poisson) processes that cause the numbers to fluctuate about the mean
and that there is no other way to determine these means and S.D.
except from the data. Under these assumptions, there should be fairly
frequent fluctuations > 1 standard deviation from the mean, and
occasional fluctuations > 2 standard deviations from the mean, but a
string of fluctuations 2 + standard deviations from the mean probably
should be looked into. (Of course, the underlying assumptions here may
be faulty, and
these bounds may need to be adjusted based on experience.)
BTW, this page seems to have blank plots, which may be due to not
enough data :
http://www.arkko.com/tools/admeasurements/stat/Eronen_Pasi-approved.html
Regards
Marshall
Jari
_______________________________________________
IETF mailing list
IETF@xxxxxxxx
https://www.ietf.org/mailman/listinfo/ietf
_______________________________________________
IETF mailing list
IETF@xxxxxxxx
https://www.ietf.org/mailman/listinfo/ietf