Hi Stephen,
On 14/01/2013 13:02, Stephen Farrell wrote:
Hi Olafur,
On 01/14/2013 04:39 PM, Olafur Gudmundsson wrote:
I have experience in process like this, as my WG DNSEXT has required
multiple implementations and inter-op testing before advancing before
advancing documents that make significant changes to the DNS protocol.
Having done this I'm confident that the resulting specifications and
code was much better.
I support this experiment but offer the following comments.
Phew, so I'm not entirely crazy, good to know:-)
Comment #1: The important part of running code is to assess
clarity of the specification, thus implementation by editors of the
document should not count as one-of-two required implementations
Implementations by editors co-workers are ok IFF the the editors keep
track of communications that lead to changes in code or draft.
First this draft only requires one implementation and not two as
your WG did. Second, I don't know what you mean by "keep track
of communications" - can you explain? (Or send a pointer to
one you used in the WG?)
If you only require one then it OUGHT to be by someone other than the
editors to count, otherwise it is just a proof of concept and we have
limited confidence that spec and code match up.
Well this can be done in number of ways,
- Issues tracker
- Postings to WG mailing list
- Notes in Appendix,
In our case we used the second one.
That's also sort of like the point Stefan W. raised. And he
suggested:
"If the source code has been developed
independently of the authoring of the draft (and ideally by non
WG participants), it is likely that the implementation and the
draft match, and that pitfalls unaware developers may find have
been found and dealt with. If, on the other hand, draft
author(s) and implementation developer(s) overlap, then it is
sensible to scrutinize the draft more closely, both with
respect to its match with the implementation and for
assumptions that author/developer may have taken for granted
which warrant documentation in the draft."
Are you saying something like that would help?
I agree with Stefan W. but would make it stronger that implementation by
editors does not count except for interoperability testing.
Personally, I think there are thousands of nuggets of advice
that we could possibly add, but I'm not convinced that we
should - I think we'd be better off trying to keep this draft
shorter and simpler and then if the experiment succeeds we can
incorporate those things that experience has shown are worth
including.
I agree, thus I suppressed lots of comments/suggestions
Comment #2: It is important that participants all realize that point of
the exercise is not to point figurers at bugs. Rather the goal is to
improve the specifications and make ALL the implementations as compliant
and bug free as possible.
Agreed. Not sure what text would change though.
Same comment about "nuggets" as above:-)
I was hoping we could work this into paragraph #3 of introduction section.
I will think about text changes and send you off-line.
Comment #3 (Section 4 point #6)
Test cases used for interoperability are critical. These test
cases MUST be public. Evaluations of test cases generated by the
implementors and/or other working group participants are critical as
that is a great indicator of the quality and thoroughness of the tests.
IMHO public test cases render the point of open vs. closed source
irrelevant.
I think that's arguable, but a reasonable argument. Again though,
I don't think we're at the point where a MUST ought go into this
draft and that'd be better done when the experiment's done.
Or do you have a suggestion for text? (I'm not at all sure how
I'd write that up now.)
I will take a stab at re-write second paragraph of section 2.1 to
reflect this but that might take a day or two.
Comment #4: The IETF-LC and WGLC statements SHOULD contain references to
the testing performed and the implementations that participated.
I could buy that as a SHOULD or "ought." I've noted that
in the working version as a change to maybe make. [1]
good.
Olafur