Hi Dave,
Thank you for your thorough review. While I accept many of your
comments, I must say I disagree with you on a few points, which together
go to the core of our motivation in writing this document. Thank you for
helping me clarify these points to myself :-)
- Our goal is much *less* ambitious than defining a framework for
documenting implementations for long-term protocol projects. Our primary
goal is to aid working group members in making informed decisions about
the quality of specifications. I believe this narrow focus is much more
realistic.
- As a result of the above: 1. We define the information as pre-RFC
only; 2. The main medium is the I-D, as opposed to Web pages/wikis/CMS
(although we see them as valid alternatives); 3. We focus on WG
documents, as opposed to "all" IETF documents.
- Despite the glowing counterexample of DKIM, it is rarely practical to
maintain implementation status information, keeping it current for years
(put bluntly, you need to pay someone to do that). The IETF certainly
does not have the processes or mechanisms to do so. And we do not try to
establish such processes here.
Please see my detailed comments below.
Thanks,
Yaron
On 2013-04-26 19:16, Dave Crocker wrote:
Given this thread on the ietf list, I guess I should forward the
following, which was done as a solicited Apps Area review for this draft:
-------- Original Message --------
Subject: Review of: draft-sheffer-running-code
Date: Wed, 24 Apr 2013 06:38:24 -0700
From: Dave Crocker <dcrocker@xxxxxxxxx>
To: draft-sheffer-running-code.all@xxxxxxxxxxxxxx,
apps-discuss@xxxxxxxx, iesg@xxxxxxxx
Howdy.
I have been selected as the Applications Area Review Team reviewer for
this draft (for background on apps-review, please see
http://www.apps.ietf.org/content/applications-area-review-team).
Please resolve these comments along with any other Last Call comments
you may receive. Please wait for direction from your document shepherd
or AD before posting a new version of the draft.
Review
------
Title: Improving Awareness of Running Code: the Implementation Status
Section
I-D: draft-sheffer-running-code-04
Reviewer: D. Crocker
Review Date: 24 April 2013
Summary:
Given an organizational motto of "Rough consensus and running code",
implementation experience has a value built into the IETF's DNA.
However formal IETF reliance on running code for specifications has been
minimal, with its only universal occurrence being in consideration of
advanced standards status; it is notably /not/ an organizational
requirement for entry-level Proposed Standard status, if only for the
very high barrier to publication this imposes on document versions that
were intended to be preliminary. The current draft proposes an
experiment to optionally include documentation of implementations as
part of document development, prior to issuance of a specification as an
RFC.
Within the broader perspective of pre- and post-standardization efforts,
it is quite common to document available implementations of a
specification. (As a convenient example, see http://dkim.org/deploy.)
In effect, the current proposal merely seeks to have a common IETF-based
means of recording a type of information that industry has already
frequently formulated.
In their current form, the draft and the proposal seem likely to be
useful. However, both could be improved.
The draft includes some language and assertions that would benefit from
being a bit more carefully written; questions, comments and suggestions
are in the detailed comments below.
The proposal itself would benefit from directly paying more attention to
existing industry practice, which is implied by the "alternatives"
section; that is, that implementation lists already happen and are
maintained after IETF document processing. The proposal should do a
better job of integrating this fact into its design.
One approach could be a relatively simple document change, to
distinguish between specifying the information to be reported, as
distinct from the means of reporting it -- think of it as payload vs.
mechanism... Specify them separately. Then define both how to
incorporate the information directly and the form of an external
citation to it; this would eliminate the document's section on
'alternatives' and treat the original and alternative methods as equal.
d/
Detailed Comments:
1. Introduction
Most IETF participants are familiar with the saying, "rough
consensus and running code" [Tao], and can identify with its
pragmatic approach. However, there are many examples of
Internet-Drafts containing protocol specification that have gone
through to publication as Proposed Standard RFCs without
implementation. Some of them may never get implemented.
The "however" implies that it is wrong or problematic for Proposed to be
assigned to unimplemented protocols. It isn't. I suggest a softer
tone, while still noting the relevantfact. Something like:
It is not a requirement of Proposed Standard status to have any
implementations; by the time of reaching that status, some
specifications have been implemented and some have not. Indeed, some
never get implemented.
Over time, a variety of policies have been implemented within the
implemented -> applied
IETF to consider running code. In the Routing Area it used to be a
requirement that one or more implementations must exist before an
Internet-Draft could be published as a Proposed Standard RFC
[RFC1264]. That RFC was later obsoleted and the requirement for
implementation was lifted, but each working group was given the
authority to impose its own implementation requirements [RFC4794]
and at least one working group (IDR) continues to require two
independent implementations.
The hypothesis behind this document is that there are benefits to
the
this -> the current
IETF standardization process of producing implementations of
protocol specifications before publication as RFCs. These benefits,
which
I suspect that that isn't its "hypothesis" at all, since the document
does nothing to specify or directly affect policies for requiring or
encouraging implementation. Perhaps there is a hope that the proposed
notation about implementations will serve as some sort of encouragement,
but that's pretty indirect; success or failure of the proposed
experiment won't validate or invalidate whether there are benefits to
early implementation. And providing for a notation convention hardly
constitutes an 'incentive'.
Rather, I believe the premise of the document is that the community is
aided by /more widely knowing about/ early implementations.
This is splitting hairs: there would be no benefit in informing the
community if there wasn't any benefit in the actual early
implementations. But fine.
include determining that the specification is comprehensible and
that there is sufficient interest to implement, are further discussed
in Section 4.
This document describes a simple process that allows authors of
process -> mechanism
Internet-Drafts to record the status of known implementations, by
including an Implementation Status section. The document defines
(quite informally) the contents of this section, to ensure that the
relevant information is included. This will allow reviewers and
working groups to assign due consideration to documents that have
the benefit of running code, by considering the running code as
evidence of valuable experimentation and feedback that has made the
implemented protocols more mature.
The above last sentence seems to be the actual claimed value proposition
for the experiment.
I agree.
This document provides a mechanism to record and publicize the
This sentence seems entirely redundant.
existence of running code. It is up to the individual working
groups to use this information as they see fit, but one result might
be the preferential treatment of documents resulting in them being
processed more rapidly.
I think it won't be the working group that 'uses' the information, but
rather IETF management and the broader community?
For most documents, AFAIK, most work is done by the WG, and a lot less
by the rest of the community. Hence the above.
The process in this document is offered as an experiment. Authors
of Internet-Drafts are encouraged to consider using the process for
their documents, and working groups are invited to think about
applying the process to all of their protocol specifications.
The scope of the intended experiment is all Internet-Drafts whether
Probably not all I-Ds, since they do not all contain implementable
specifications.
Sheffer & Farrel Expires October 14, 2013 [Page
3] Internet-Draft Running Code
April 2013
produced within IETF working groups, outside working groups but
What I-Ds are produced by "outside working groups"? I can't tell who
this refers to.It's probably just as well to remove the attempted case
analysis for groups and just say that the scope is all specifications
issued as I-Ds.
In fact others have commented that we should exclude individual
submissions for process reasons.
intended for IETF consensus, or for publication on the Independent
Stream. However, it is expected that most benefit from the
that most benefit from the experiment will
->
the greatest benefit in the experiment will
experiment will be seen with Standards Track documents developed
within working groups.
Here's my guess:
1. Those offering comments or making decisions about the status of
document will be aided by this added information.
2. Those deciding about whether to write code or deploy it will be aided.
I think the latter is likely to be the greater benefit.
The authors of this document intend to collate experiences with this
experiment and to report them to the community.
2. The "Implementation Status" Section
Each Internet-Draft may contain a section entitled "Implementation
Status". This section, if it appears, should be located just before
the "Security Considerations" section and contain, for each existing
implementation:
o The implementation's name and/or a link to a web page describing
the implementation. o A brief general description. o The
implementation's level of maturity: research, prototype, alpha, beta,
production, widely used etc. o Coverage: which parts of the protocol
specification are implemented and which versions of the
Internet-Draft were implemented. o Licensing: the terms under which
the implementation can be used. For example: proprietary, royalty
licensing, freely distributable with acknowledgement (BSD style),
freely distributable with requirement to redistribute source (GPL
style), other (specify). o Contact information: ideally a person's
name and email address, but possibly just a URL or mailing list.
List the organization that did the implementation and contact
information for the proper place in the organization.
In addition, this section can contain information about the
interoperability of any or all of the implementations.
Since this information is necessarily time-dependent, it is
inappropriate for inclusion in a published RFC. The authors should
Way to bury the lead. This stuff won't be in the published document?
It's only for pre-publication?
Very clearly YES.
include a note to the RFC Editor requesting that the section be
removed before publication.
2.1. Introductory Text
The following boilerplate text is proposed to head the
Implementation Status section:
Sheffer & Farrel Expires October 14, 2013 [Page
4] Internet-Draft Running Code
April 2013
This section records the status of known implementations of the
protocol defined by this specification at the time of posting of this
Internet-Draft, and is based on a proposal described in [RFC Editor:
replace by a reference to this document]. According to [RFC Editor:
replace by a reference to this document], "this will allow reviewers
and working groups to assign due consideration to documents that have
the benefit of running code, by considering the running code as
evidence of valuable experimentation and feedback that has made the
implemented protocols more mature. It is up to the individual
working groups to use this information as they see fit".
Authors are requested to add a note to the RFC Editor at the top of
this section, advising the Editor to remove the entire section
before publication, as well as the reference to [RFC Editor: replace
by a reference to this document].
3. Alternative Formats
Yes!
Sometimes it can be advantageous to publish the implementation
status separately from the base Internet-Draft, e.g. on the IETF
wiki:
o When the Implementation Status section becomes too large to be
conveniently managed within the document. o When a working group
decides to have implementors, rather than authors, keep the status of
their implementations current. o When a working group already
maintains an active wiki and prefers to use it for this purpose.
It is highly desirable for all readers of the Internet-Draft to be
made aware of this information. Initially this can be done by
replacing the Implementation Status section's contents with a URL
pointing to the wiki. Later, the IETF Tools may support this
functionality, e.g. by including such a link from the HTMLized draft
version, similar to the IPR link.
If the implementation status is published separately from the I-D,
then this information needs to be openly available without requiring
authentication, registration or access controls if it is to have any
useful effects.
4. Benefits
Publishing the information about implementations provides the
working group with several benefits:
Sheffer & Farrel Expires October 14, 2013 [Page
5] Internet-Draft Running Code
April 2013
o Participants are motivated to implement protocol proposals, which
helps in discovering protocol flaws at an early stage.
The psychological premise seems to be that getting cited in this list
will somehow serve as an incentive to implementers. I can imagine that
they'll like being listed, but find it extremely difficult to believe
that it will affect whether they do the work; not that I think it's a
deciding factor, but note that the ego value is further reduced by the
information's being ephemeral, since it it removed prior to RFC
publication!
Actually not (subtly so). The premise is making this information public
and an explicit part of WG deliberations will incentivize
implementations (i.e. not the ego factor at all).
o Other participants can use the software, to evaluate the
usefulness of protocol features, its correctness (to some degree) and
other properties, such as resilience and scalability.
Efficacy of the spec.
o WG members may choose to perform interoperability testing with
known implementations, especially when they are publicly available.
Interoperability of the spec.
o In the case of open source, people may want to study the code to
better understand the protocol and its limitations, determine if the
implementation matches the protocol specification, and whether the
protocol specification has omissions or ambiguities.
Reference implementation as exemplar.
o Working group chairs and ADs may use the information provided to
help prioritize the progress of I-Ds in some way.
What does "prioritize the progress" mean?
When there are multiple competing proposals to solve some problem.
o Similarly, the information is useful when deciding whether the
document should be progressed on a different track (individual
submission, Experimental etc.).
Assess spec maturity. (This seems redundant with the combination of
Efficacy and Interoperability?)
o And lastly, some protocol features may be hard to understand, and
for such features, the mere assurance that they can be implemented is
beneficial.
mumble. Might highlight problems with the spec language. Might
undermine the spec text by causing code to be used in place of it?
I'm with you on both mumbles.
We do not specify here whether and to what degree working groups are
expected to prefer proposals that have "running code" associated
with them, over others that do not.
5. Process Experiment
The current proposal is proposed as an experiment. The inclusion of
"Implementation Status" sections in Internet-Drafts is not
mandatory, but the authors of this document wish to encourage authors
of other Internet-Drafts to try out this simple process to discover
whether it is useful. Working group chairs are invited to suggest
this process to document editors in their working groups, and to draw
the attention of their working group participants to "Implementation
Status" sections where they exist.
Following a community discussion, it was concluded that [RFC3933] is
not an appropriate framework for this experiment, primarily because
no change is required to any existing process.
It defines a document meta-data encoding mechanism. That's not really a
"process".
7. Security Considerations
This is a process document and therefore, it does not have a direct
effect on the security of any particular IETF protocol. Better
Better -> However, better
d/