On 01/28/2013 04:27 AM, Joe Touch wrote:
> About the idea of an "experiment":
Right. The context being its an RFC 3933 IETF process
experiment.
>
> On 1/25/2013 5:07 AM, Stephen Farrell wrote:
>>
>> Responses to some points below but I'd really like to ask
>> people to consider a few things here:
>>
>> - what's proposed is an experiment, it'd likely get tried out
>> a few times and won't consume any huge resource anywhere
>
> If this is an experiment, then you presumably answers to the following
> questions:
>
> 1- what is your an hypothesis?
> 2- what you intend to measure?
> 3- what is your 'control' against which to compare the results?
> 4- what is your objective metric for success/failure?
Well, its not a scientific experiment, and nor does it need to
be. Quoting 3933:
- " A statement of the problem expected to be resolved is
desirable but not required (the intent is to keep the firm
requirements for such an experiment as lightweight as possible).
Similarly, specific experimental or evaluative criteria, although
highly desirable, are not required -- for some of the process
changes we anticipate, having the IESG reach a conclusion at the
end of the sunset period that the community generally believes
things to be better (or worse) will be both adequate and
sufficient. "
My take is that even though 3933 says "desirable" a couple of
times there, it's probably best to not go there, at least in this
case, but for now probably more generally. The reason I think
that is that I reckon the IETF has got itself into a log-jam
that almost prevents us from modifying our processes in any way
and every additional word provides additional barriers to doing
anything, since there'll always be someone for whom those added
words raise what seems to them to be a red flag.
I do agree that's far from ideal, but so is our current
process-change log-jam. (BTW, I already said this a few months
ago in response to a similar comment on I think -00 or -01.)
Separately, I don't believe this proposed experiment does have
any objective criteria on which we could get consensus and that
would be practical to measure - I think the outcome is more
likely to be subjective, (e.g. "seemed to help because of X")
and that the follow-up, if any, would require further
consideration - my guess is that if the experiment were some
form of success then we'd need to start over on documenting
the thing(s) that might be worth including in an update to
BCP9, but then that'd be based on some experience with
running the experiment rather than on your or my speculation
as to what might or might not work well or badly.
>
> I've heard only one hypothesis - that this reduces time to publication.
> I disagree that this is a useful hypothesis to test for the following
> reasons:
>
> - time to publication isn't a goal of the IETF
> IMO, any doc that isn't useful in 5 years ought
> to not be published here; we don't need to
> document every sneeze
IMO reduced time to publication is definitely *a* goal.
I've heard lots of IETF participants complain about how
long things take, lots of times. Perhaps you're in the
rough on that. (I also note that this experiment doesn't
actually aim for any major reduction in time to publication
as it says in the draft.)
The draft itself also discusses reasons why running code
might also lead to better quality specifications.
> - thorough review ought to be a requirement
> and this 'experiment' potentially compromises that
> by reducing the overall time of review
I think the liklihood of that doing damage during the
experiment is very small. In addition, I might be wrong,
but the thoroughness of review doesn't correlate that
well with the duration of review would be my guess. While
secdir reviews are pretty variable, they are almost all
done in a tight timeframe and a good number of them are
very good. Same for gen-art and apps-dir reviews. So
we do have evidence that good review can be done in a
short timeframe. Equally, I've seen quite a few drafts
that have reached the IESG after years that still
have quite a lot that needs to be fixed and that a
thorough review ought have caught.
> - community resources ought to be considered
> and this 'experiment' burns group resources
> due to having a broad group concurrently review
> a doc that could have been reviewed by smaller
> groups first
Yes, this experiment should bring a few drafts to IETF
review earlier. The WG chairs are still supposed to
only send a publication request when they're satisfied
the draft represents the rough consensus of the WG.
However, I'm looking at changing some of the wording
on this in response to John's point about having two
parallel discussions. I'll post to the list when I've
had a chance to write that up in the working version.
>
> Given the limited cycles this community has to review docs, I cannot see
> a benefit to this experiment that is worth the cost.
Fair enough. We disagree.
> Having this entire community burn cycles on this document speaks for
> itself. It should have been vetted in a smaller, more invested community
> first.
I'm following the 3933 process.
I don't know of any smaller but open venue for discussing rfc
3933 experiments. Perhaps there ought be such, but given how
rarely 3933 experiments are proposed, that'd be a very quiet
list.
In any case I would confidently predict that any process change
or experiment will generate this much traffic during IETF
last call so I'm not sure there's a saving;-) If someone manages
to get any kind of process change approved without a major flare
up on this list then I'll happily buy them a beer to celebrate.
If you'd rather we make 3933 historic or replace it with
something else, that's a different topic.
> Calling something an 'experiment' doesn't make it worthwhile to test.
I believe this draft does meet all the requirements of
3933 and is an experiment in that sense. I agree that not all
such things would be worthwhile to test, but a number of
people have agreed that this was worthwhile to test. Whether
or not that means we ought go forward with this is a call
for Adrian and the IESG at the end of IETF LC.
S.
>
> Joe
>
>
>
>