> On Oct 28, 2017, at 9:51 AM, Sam Hartman <hartmans-ietf@xxxxxxx> wrote: > >>>>>> "Russ" == Russ Housley <housley@xxxxxxxxxxxx> writes: > >>> Some of us were very badly burned, in one way or another, by >>> formal conformance tests of OSI implementations the best part of >>> 30 years ago. So while I fully support Michael on "facilitating >>> conformance testing" and would even insert the word "rigorous", I >>> would be very cautious about "formal methods" (except for things >>> like MIB modules and YANG, where clearly a formal check is >>> required). >>> >>> Interoperability remains, IMHO, much more important than formal >>> correctness. Implementations can be formally correct but faulty >>> in practice. >>> >>> In any case, I don't think that distinction is relevant to the >>> ISOC mission, which needs to retain some level of abstraction. > > Russ> Care must be taken that ISOC does not create an unfortunate > Russ> feedback loop by sponsoring standards development and > Russ> conformance testing of implementations. Testing should > Russ> provide feedback into the standards process, but testing > Russ> should not drive the standards process. > > Russ, I suspect I probably agree with you. However, I spent a couple of > minutes trying to understand what you wrote and realized that I was > having a hard time. Could you give a couple of example of giving > feedback into the standards process (what you'd like to see) and testing > driving the standards process so I can see the difference? I am aware of a situation in another standards organization where the organization that wrote the standards also did the testing. The testing group wanted to have some aspects of the standards changed (make it easier to conform) so that the testers could generate more revenue. The testers we being measured on the revenue, not making the ecosystem better. Russ