On 6/14/2017 12:56 PM, heasley wrote: > Tue, Jun 13, 2017 at 03:03:07PM -0700, Joe Touch: >> Hi, all, >> >> ... >>> Title : The Harmful Consequences of >>> Postel's Maxim >>> https://tools.ietf.org/html/draft-thomson-postel-was-wrong-01 >> I completely agree with John Klensin that a test suite defines the >> protocol standard (warts and all). > This may be, even if a pair of implementations were required company, > but is that wholly negative? Not necessarily, but it does negate the need for a specification. "A designer with one set of requirements always knows what to follow; a designer with two is never sure" (a variant of "a person with one watch always knows what time it is...") If you have a test suite, that singularly defines the protocol to the exclusion of the spec, otherwise you don't have a complete suite. > The test suite is not stuck in time; it can > evolve to test things that its authors had not anticipated, its own bugs, > or evolution of the protocol itself. It is also more likely to catch bugs > and protocol design flaws than interoperability testing alone. Interoperability tests are just a variant of test suites - just that they are all relative to each other, rather than to a single entity. However, forcing testing to a single entity will necessarily catch less issues than cross-testing can. > At the very least, an open test suite allows everyone to test against a > know baseline, before they test interoperation with other implementations. > For example, a rigid test suite that fails when an reserved field is not > its prescribed value, would be an asset for future development that > allocates the field. That then begs the question: - what happens if my version passes the test suite but fails interoperability tests? > Certainly the mentioned JSON problems would have benefitted from a test > suite and interoperability testing. But what happens when they provide conflicting results? At that point, you need to decide which one wins; if that wins, why do the other? > Clearly, peer review and > interoperability testing are insufficient. Nothing is ever sufficient unless it is complete, and complete protocol tests are not feasible for anything other than toy examples. The state space blows up. Joe