On 1/7/21 5:30 AM, Hannes Tschofenig wrote:
Hi Keith,
I certainly agree that we can't just wait for random outsiders to review things, and I appreciate the hard work it takes to get wide review. But I've also seen high-quality and useful feedback from the extended IETF community, and I don't think we should be adopting practices that discourage that.
From your experience, what are these reviews typically based on? Are they based on drafts or something else? If I recall great feedback I received "recently" then the review from Bob Briscoe on the SUIT architecture comes to mind (which happened late in the process based on the draft) and feedback from implementers (which was, of course, also based indirectly on a draft version).
I think the feedback is as likely to be based on f2f conversations and
email conversations and also WG presentations, in addition to drafts.
More realistically, as work progresses, early feedback is more likely to
be based on such conversations, and later feedback more likely to be
based on drafts. But conversations play a large role even fairly late
in the process.
(I often find the WG presentation materials to be MUCH better at
facilitating useful conversations than the drafts, even if I find time
spent "presenting" those materials to be largely wasted. Being able to
read those materials without sitting through the presentations is
great. Being able to have real time conversations about those materials
is also great if people have read those materials in advance.)
I think it's worth pointing out that reviewing documents is itself
difficult and tedious work, and often it's easier to build consensus
around design decisions before picking apart and wordsmithing the
documents. But it's also the case that people usually won't take a
proposal seriously until there's a reasonably complete draft of it. So
you write a draft for people to attack (which should be complete enough
to look credible but still leave some decisions for people to haggle
over), then you have conversations, then you rewrite the draft - often
substantially and multiple times, until finally you converge enough to
do wordsmithing. And it's in this last stage that I see github as
being a useful though still really awkward tool. But the conversations
continue throughout.
Looking at this now, I wonder if one of the problems with our typical
process is that we insist on having even the earliest proposals look
like an RFC, as opposed to say, a set of bullet points.
Some of the early feedback I would not call "review" but something closer "feed forward". WGs and document authors need to understand the constraints of the space they are designing for, and input from outside the WG core can be very valuable for this and should be encouraged.
This is also an interesting case because it is input (for feed forward, as you call it) that is substantially different from the feedback later in the life of a specification (or set of specifications). In my experience this type of input is given during BOFs, in informal discussions, etc. and is often extremely time consuming for the persons providing the input. What is your experience there?
In my experience the feedback occurs not so much in BOFs (though this
does happen) as in informal discussions and WG discussions.
IMO most constructive feedback, regardless of when it's given or the
form it takes, is extremely time consuming for the persons providing the
input. But the hardest part of the work isn't in writing the words;
it's in understanding what's really important to different people with
different experiences, different points of view, different ways of
expressing themselves.
I am wondering whether the discussion about feedback / input isn't largely orthogonal from the question what tools (email, Github, Slack, etc.) are being used. Providing input early in the life of a specification has always been a challenge - even before Github was around.
I think the tools have an effect at every stage. Maybe no tool will
make the process of understanding easier, but poor tools can certainly
make the process more difficult.
But I also think that the work habits we have, many of them well
entrenched in IETF culture, may also be making the process much more
difficult than it needs to be.
Keith
